Optimized deduplication based on backup frequency in a distributed data storage system

ABSTRACT

Disclosed deduplication techniques at a distributed data storage system guarantee that space reclamation will not affect deduplicated data integrity even without perfect synchronization between components. By understanding certain “behavioral” characteristics and schedule cadences of backup operations that generate backup copies received at the distributed data storage system, data blocks that are not re-written by subsequent backup copies are pro-actively aged, while promoting continued retention of data blocks that are re-written. An expiry scheme operates with block-level granularity. Each unique deduplicated data block is given an expiry timeframe based on the block&#39;s arrival time at the distributed data storage system (i.e., when a backup copy supplies the block) and further based on backup frequencies of the various virtual disks referencing a unique system-wide identifier of the block, which is based on the block&#39;s hash value. Communications between components are kept to an as-needed basis. Cloud-based and multi-cloud configurations are disclosed.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.17/153,667 filed on Jan. 20, 2021, which claims priority to U.S. Pat.App. No. 63/070,162 filed on Aug. 25, 2020. Any and all applications forwhich a foreign or domestic priority claim is identified in theApplication Data Sheet of the present application are herebyincorporated by reference in their entireties under 37 CFR 1.57.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentand/or the patent disclosure as it appears in the United States Patentand Trademark Office patent file and/or records, but otherwise reservesall copyrights whatsoever.

BACKGROUND

Deduplication in a distributed data storage platform requires tightlycoupled communications among components to ensure that deduplicationtracking is kept current at the various nodes that form the storageplatform. However, communicating a lot of information frequently amongmany components places a substantial burden on network bandwidth andcomponent cycle time that is needed for processing data storage andretrieval requests. Therefore, a solution is needed that scales well tolarge and very active data storage platforms while maintaining highlyaccurate distributed deduplication tracking with low communicationsoverhead.

SUMMARY

The present inventors devised a technological solution that optimizesdeduplicated storage of backup copies at a distributed data storageplatform (hereinafter the “distributed data storage system”). Thepresent solution is scalable and guarantees that space reclamation willnot affect deduplicated data integrity even without perfectsynchronization between components. The illustrative approach balancesthe need for aggressive space reclamation of stale data against the needto minimize re-writes of existing deduplicated data. Discarding data tooaggressively will slow system performance as new data needs to bewritten that was already on the distributed data storage system.Conversely, failing to discard stale data reduces the usable storage ofthe distributed data storage system.

By understanding certain “behavioral” characteristics and schedulecadences of backup operations that generate backup copies received atthe distributed data storage system, the present approach pro-activelyages data blocks that are not re-written by subsequent backup copies,while promoting continued retention of data blocks that are re-written.Backup copies are typically generated on a substantially regularschedule, e.g., weekly full backups followed by daily incrementalbackups. Thus, the contents of full backup copies effectively replaceearlier full and incremental backup copies, e.g., on a weekly basis.Since the illustrative distributed data storage system is an append-onlysystem, unique data blocks that are not supplied again by later backupcopies become stale after several full backup cycles, because the sourcedata being backed up has changed and generates different backup datablocks. Eventually, older backup copies will become stale and the backupsystem that generated them will prune them from the distributed datastorage system. At this point, stale data blocks that are no longerreferenced by any backup copies are pro-actively deleted (“garbagecollected”) from the distributed data storage system.

The present inventors devised an expiry scheme that operates atblock-level granularity on the distributed data storage system. Thisapproach advantageously overcomes some prior-art deficiencies in which asingle data block referenced by a single virtual disk would preventdiscarding all other data blocks associated with the entire virtualdisk, thus retaining a lot of stale data. This prior-art granularityscheme operated at the virtual disk level. The present approach alsoovercomes other prior-art deficiencies in which a new deduplicationstore was opened periodically, which automatically expired data blocksin the preceding deduplication store, but disadvantageously requiredre-writes of retained older data blocks already in the system.

Here, in contrast to these prior-art solutions, each unique deduplicateddata block is given an expiry timeframe based on the block's arrivaltime at the distributed data storage system (i.e., when a backup copysupplies the data block) and further based on backup frequencies of thevarious virtual disks referencing the data block. The present solutionincludes a global (or system-wide) deduplication repository, which isconfigured as a virtual disk that is partitioned and replicated acrossstorage nodes. The global system deduplication virtual disk (or“system-wide deduplication virtual disk) is not exposed as a storagetarget to the backup system generating the backup copies and is managedas a strictly internal resource of the distributed data storage system.By using the system-wide deduplication virtual disk, the distributeddata storage system maximizes deduplication ratios across data sources,i.e., regardless of the user virtual disk addressed by an incomingbackup copy. However, different user virtual disks may have differentbackup schedules. Therefore, the illustrative block-level expiryapproach considers the various backup schedules in assigning expirytimeframes and deciding when and whether to discard individual datablocks.

A periodic “garbage collection” or “GC” or discard cycle evaluates eachdeduplicated data block tracked by the distributed data storage system.Blocks “written” (i.e., supplied by an incoming backup copy) in thepreceding cycle are analyzed for reference counting and expirytimeframes. Illustratively, each backup copy received by the distributeddata storage system comprises one or more files, and the distributeddata storage system may address each file to a distinct user virtualdisk. When the backup system prunes stale backup copies from thedistributed data storage system, the pruning causes the distributed datastorage system to logically delete the various user virtual diskscomprising the backup files associated with the particular backup copy.

Because of deduplication, not every block that comes in with a backupcopy is actually added to the distributed data storage system, but thewrite request is noted and tracked. A new data block with a new hashvalue receives a unique system-wide identifier (e.g., the “dedupe blockidentifier” or “DDblockID”) and the new data block is added to theglobal deduplication virtual disk. A block that is “written” one or moretimes within a preceding cycle will have its expiry extended at leastone more cycle during the analysis performed by the discard cycle.Should subsequent “writes” continue, the block's expiry will be extendedagain, thus ensuring that the block stays in the global deduplicationvirtual disk because it is still current. But when the subsequent writeswane, the block will eventually expire. If there are no more backupcopies (and corresponding user virtual disks) on the distributed datastorage system that reference the particular data block, the discardcycle identifies the data block for deletion, and a compaction enginewill delete the block from the distributed data storage system. Shouldthe expired data block appear again in a later write request, it will beseen as a new block and written anew to the distributed data storagesystem.

One of the key aspects of the disclosed approach is that in each discardcycle, the expiry timeframe of a deduplicated data block (DDblockID) maybe extended to accommodate the least frequent full-backup frequency ofany user virtual disk that references the data block. In this way, allvirtual disks referencing the DDblockID are guaranteed the data block'ssurvival at least through the next full backup operation. This keyaspect provides flexibility and scalability without compromising dataintegrity. Moreover, this key aspect maintains the block-levelgranularity of the disclosed expiry scheme, which improves theeffectiveness of the discard process.

The illustrative distributed data storage system comprises storageproxies that intercept backup copies being written to the distributeddata storage system. Rather than adding every incoming data block to thedistributed data storage system, the storage proxy applies adeduplication routine to the data block. The storage proxy uses hashing(e.g., MD5 without limitation) to compute a hash value for the incomingdata block and checks whether the hash value is present in a local indexor data structure at the storage proxy (e.g., DDCache).

If the storage proxy does not find the hash value in DDCache, thestorage proxy passes the hash value to a metadata subsystem on a storageservice node and receives an update therefrom, comprising a DDblockIDand expiry timeframe if the DDblockID is known, i.e., has beenpreviously added to the system-wide deduplication virtual disk. If thehash value is new to the metadata subsystem or if the expiry timeframefor the DDblockID is in the past (i.e., expired DDblockID according tothe metadata subsystem), the data block is treated as a new block, andthe storage proxy will store the data block to the system-widededuplication virtual disk. The system-wide deduplication virtual diskis partitioned and duplicated across a plurality of data storagesubsystems that are distinct from the metadata subsystem. The datastorage subsystem assigns a new DDblockID to the new data block. Themetadata subsystem updates its own tracking data structures, assigns afuture expiry timeframe (“epoch”) to the new DDblockID, and transmitsthe DDblockID and its expiry epoch to the storage proxy for updating itsown local index (e.g., DDCache). If it turns out that the metadatasubsystem reported to the storage proxy that the DDblockID correspondingto the hash value is not expired, the storage proxy updates its DDCacheand the write request is noted in data structures at the metadatasubsystem.

Otherwise, if the storage proxy found the hash value of the data blockwithin its own index (e.g., DDCache), the storage proxy checks theexpiry epoch of the corresponding DDblockID. If the DDblockID is notexpired, the storage proxy reports the write request to the metadatasubsystem so that it can update its local tracking data structures. Onthe other hand, if the storage proxy determines that the DDblockID hasan expired epoch (according to DDCache), the storage proxy submits thehash value to the metadata subsystem as if not found in DDCache asdescribed above.

Thus, during the I/O cycle of a given data block, its hash value iscalculated, its presence in the global deduplication virtual disk isdetermined, and its pre-existing expiry (if any) determines whether thedata block is added to the global deduplication virtual disk. The indexat the storage proxy (e.g., DDCache) is updated only as needed tominimize updates sent by the metadata subsystem; no attempt is made tokeep DDCache fully synchronized with the tracking data structures at themetadata subsystem. The metadata subsystem updates certain localtracking data structures for every incoming write request. One of thesedata structures is the illustrative DDTracker column family, which isupdated for every write request from an incoming backup copy and enablesproper reference counting.

Illustratively, a discard cycle runs weekly based on the typical cadenceof full backup cycles, but the invention is not so limited. During afirst phase of the discard cycle (the preparation stage), the metadatasubsystem scans each DDTracker column family and adds the resultingprocessed information to a persistent column family used for trackingDDblockID reference counts (e.g., DDRefCount). For example, DDblockID_7that was “written” or addressed to user virtual disk 1 (e.g.,uservdisk_1) is added to DDRefCount showing that uservdisk_1 “wrote”DDblockID_7 to the system during an epoch as indicated by DDTracker.DDRefCount assigns an expiry to this DDblockID_7 instance based on thefull backup frequency of uservdisk_1. Likewise, another data blockhaving the same DDblockID (e.g., DDblockID_7), which was “written” ortargeted to another user virtual disk (e.g., uservdisk_N) is added toDDRefCount showing that uservdisk_N “wrote” DDblockID_7 to the systemduring a write epoch as indicated by DDTracker. DDRefCount assigns anexpiry to this other DDblockID_7 instance based on the full backupfrequency of uservdisk_N, which may differ from that of uservdisk_1.Thus, every undeleted DDblockID in the global deduplication virtual diskhas entries (columns) in DDRefCount corresponding to the various epochswhen a new instance of the data block entered the system and columns foruser virtual disks referencing the DDblockID. A maximum value of thevarious expiry columns, i.e., the longest expiry timeframe assigned tothe DDblockID in the system (e.g., Eu) ensures that the DDblockID willsurvive between the sparsest virtual disk backups, and later helps todetermine whether a certain DDblockID should be discarded.

Once the preparation stage has fully processed all DDTracker columnfamilies and persisted DDRefCount, a second phase of the discard cyclebegins—the “garbage collection” or discard stage. The second phasechecks whether the user virtual disks in DDRefCount are still present onthe distributed data storage system. When a backup copy is pruned by thebackup system that created it, the virtual disk(s) created to store thebackup copy and/or its constituent files are deleted from thedistributed data storage system configuration. Thus, backup copydeletions may result in DDRefCount decrementing the reference counts ofcertain DDblockIDs referenced by the corresponding virtual disks. Thesecond phase further checks the maximum expiry epoch (e.g., Eu) of everyDDblockID in DDRefCount. DDblockIDs with zero reference counts andexpired Eu are placed by the metadata subsystem on discard lists.Because the global system deduplication virtual disk is distributedamong a plurality of different storage containers on different storageservice nodes, the discard lists are segregated by storage container andtransmitted to the data storage subsystem nodes hosting the respectivestorage containers. There, the resident compaction engine ensures thatthe actual data blocks corresponding to the DDblockID are discarded,thus freeing up storage space.

On the other hand, a DDblockID that was “written” after the last discardcycle, is given an extension of time. Accordingly, the expiry epoch isincremented by the full backup frequency of the user virtual diskassociated with the write request. If the maximum expiry epoch (Eu)increases at this point, the increase is reflected in the local index ofthe metadata subsystem (e.g., DDInfo). In this way, subsequent writerequests of this DDblockID will be evaluated with respect to the updated(later) expiry epoch, thus extending the lifetime of the DDblockID onthe distributed data storage system based on its being recently“re-written.”

One of the key advantages of the disclosed optimized deduplicationscheme is that it continues to operate throughout the distributed datastorage system even if storage proxies and/or metadata-hosting nodes aredown. Thus, storage proxies and/or metadata nodes that are down cannotaffect the expiry-based aging of deduplicated data blocks going on inthe rest of the system. The present solution guarantees that stalereferences to DDblockIDs that may be lingering in these non-functionalcomponents cannot prevent a particular DDblockID from being discarded.When a storage proxy or metadata subsystem revives after one or morediscard cycles, its local index (e.g., DDCache, DDInfo) indicates thatmany, if not all, incoming data blocks are expired, because expiryepochs have not been updated while down. This triggers a check-in withan operational metadata node as explained above. The check-in sets therecord straight according to current expiry information at the workingmetadata node, handles the data block appropriately (e.g., new add vs.deduplicated), and provides an update to the newly revived component ifneeded. This scheme guarantees that stale data blocks will not bemistakenly referenced by components that revive with out-of-datetracking indexes. This aspect provides resiliency to the distributeddata storage system without preventing space reclamation.

In sum, the present application discloses a technological improvementthat enables system-wide deduplication with block-level expirygranularity. The useful life of each deduplicated data block is based onexpiry parameters that relate to backup frequencies of the virtual disksreferencing the data block, thus guaranteeing that data blocks are keptaround between full backup cycles and are extended if still current.Blocks are retained as long as needed to bridge the gap between sparserbackup operations. Tracking data structures are updated only as needed,thus saving processing cycles and network bandwidth. Moreover, thepresent solution guarantees that stale references to DDblockIDslingering in non-functional components cannot dictate whether aparticular DDblockID is discarded.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram depicting a distributed data storage system100 according to an illustrative embodiment.

FIG. 1B is a block diagram illustrating some details of distributed datastorage system 100 comprising separately scalable storage service nodesaccording to an illustrative embodiment.

FIG. 1C is a block diagram depicting certain subsystems of the storageservice of distributed data storage system 100, according to anillustrative embodiment.

FIG. 1D is a block diagram depicting a virtual disk distributed across aplurality of storage service nodes and also depicting a plurality ofstorage resources available at each storage service node according to anillustrative embodiment.

FIG. 1E is a block diagram depicting a typical I/O workflow for writeoperations originating with an application.

FIG. 1F is a block diagram depicting a typical I/O workflow for readoperations originating with an application.

FIG. 1G is a block diagram illustrating a backup configuration thatdepicts an illustrative data storage management system using distributeddata storage system 100 for storing secondary (backup) copies accordingto an illustrative embodiment.

FIG. 1H is a block diagram illustrating a backup configuration in whicha third-party backup system uses the illustrative distributed datastorage system 100 for storing secondary (backup) copies according to anillustrative embodiment.

FIG. 2A is a block diagram depicting components that play a role in thepresent deduplication solution according to an illustrative embodiment.

FIG. 2B is a block diagram depicting components that play a role in thepresent deduplication solution according to an illustrative embodiment.

FIG. 2C depicts an illustrative embodiment of a column family (e.g.,DDCache, DDInfo) for associating hash values with correspondingDDblockID and associated expiry epoch.

FIG. 2D depicts an illustrative column family for tracking writerequests received by storage proxies, e.g., DDTracker.

FIG. 2E depicts an illustrative column family that is updated duringeach discard preparation stage based at least in part on scanning andprocessing DDTracker information, e.g., DDRefCount.

FIG. 2F depicts an illustrative column family that assigns the workloadof scanning DDTracker column families for the purpose of updatingDDRefCount.

FIG. 2G depicts an illustrative column family that comprises DDblockIDdiscard lists.

FIG. 3 depicts a fence diagram that illustrates some salient operationsoccurring during an I/O cycle of an incoming data block intercepted by astorage proxy.

FIG. 4 depicts some salient operations of a method 400 according to anillustrative embodiment.

FIG. 5 depicts some salient operations of block 408 in method 400according to an illustrative embodiment.

FIG. 6 depicts some salient operations of a method 600 according to anillustrative embodiment.

FIG. 7 depicts some salient operations of block 606 in method 600according to an illustrative embodiment.

DETAILED DESCRIPTION

Detailed descriptions and examples of systems and methods according toone or more illustrative embodiments of the present invention may befound in the section entitled OPTIMIZED DEDUPLICATION BASED ON BACKUPFREQUENCY IN A DISTRIBUTED DATA STORAGE SYSTEM, as well as in thesection entitled Example Embodiments, and also in FIGS. 1G-7 herein.Furthermore, components and functionality for optimizing deduplicationin a distributed data storage system may be configured and/orincorporated into the distributed data storage system described hereinin FIGS. 1A-1F.

Various embodiments described herein are intimately tied to, enabled by,and would not exist except for, computer technology. For example,hashing, analyzing, and transmitting data among the disclosed componentsdescribed herein cannot reasonably be performed by humans alone, withoutthe computer technology upon which they are implemented.

Generally, the systems and associated components described herein may becompatible with and/or provide at least some of the functionality of thesystems and corresponding components described in one or more of thefollowing U.S. patents and patent applications assigned to CommvaultSystems, Inc., each of which is hereby incorporated by reference in itsentirety herein.

Title USPTO Ser. No. U.S. Pat. No. Filing Date STORAGE SYSTEM FOR14/322,813 10,067,722 Jul. 2, 2014 PROVISIONING AND STORING DATA TO AVIRTUAL DISK METHOD FOR WRITING DATA TO A 14/322,832 9,875,063 Jul. 2,2014 VIRTUAL DISK USING A CONTROLLER VIRTUAL MACHINE AND DIFFERENTSTORAGE AND COMMUNICATION PROTOCOLS DISK FAILURE RECOVERY FOR 14/322,8509,424,151 Jul. 2, 2014 VIRTUAL DISK WITH POLICIES CREATING AND REVERTINGTO A 14/322,855 9,558,085 Jul. 2, 2014 SNAPSHOT OF A VIRTUAL DISKCLONING A VIRTUAL DISK IN A 14/322,867 9,798,489 Jul. 2, 2014 STORAGEPLATFORM WRITING TO A STORAGE 14/322,868 9,483,205 Jul. 2, 2014 PLATFORMINCLUDING A PLURALITY OF STORAGE CLUSTERS TIME STAMP GENERATION FOR14/322,871 9,411,534 Jul. 2, 2014 VIRTUAL DISKS METHOD FOR WRITING DATATO 14/684,086 9,864,530 Apr. 10, 2015 VIRTUAL DISK USING A CONTROLLERVIRTUAL MACHINE AND DIFFERENT STORAGE AND COMMUNICATION PROTOCOLS ON ASINGLE STORAGE PLATFORM DYNAMICALLY SPLITTING A RANGE 14/723,380Abandoned May 27, 2015 OF A NODE IN A DISTRIBUTED HASH TABLE STORAGESYSTEM WITH VIRTUAL PCT/US2015/38687 Expired Jun. 30, 2015 DISKS GLOBALDE-DUPLICATION OF 15/155,838 10,846,024 May 16, 2016 VIRTUAL DISKS IN ASTORAGE PLATFORM DE-DUPLICATION OF CLIENT-SIDE 15/156,015 10,795,577 May16, 2016 DATA CACHE FOR VIRTUAL DISKS PERSISTENT RESERVATIONS FOR15/163,446 10,248,174 May 24, 2016 VIRTUAL DISK USING MULTIPLE TARGETSSYNCHRONIZATION OF METADATA 15/834,921 10,740,300 Dec. 7, 2017 IN ADISTRIBUTED STORAGE SYSTEM IN-FLIGHT DATA 15/912,374 10,848,468 Mar. 5,2018 ENCRYPTION/DECRYPTION FOR A DISTRIBUTED STORAGE PLATFORM PERSISTENTRESERVATIONS FOR 16/274,014 10,691,187 Feb. 12, 2019 VIRTUAL DISK USINGMULTIPLE TARGETS DISTRIBUTED DATA STORAGE 63/053,414 Jul. 17, 2020SYSTEM USING ERASURE CODING ON STORAGE NODES FEWER THAN DATA PLUS PARITYFRAGMENTS DISTRIBUTED DATA STORAGE 63/065,722 Aug. 14, 2020 SYSTEM USINGERASURE CODING ON STORAGE NODES FEWER THAN DATA PLUS PARITY FRAGMENTSOPTIMIZED DEDUPLICATION BASED 63/070,162 Aug. 25, 2020 ON BACKUPFREQUENCY IN A DISTRIBUTED DATA STORAGE SYSTEM ANTI-ENTROPY-BASEDMETADATA 63/081,503 Sep. 22, 2020 RECOVERY IN A STRONGLY CONSISTENTDISTRIBUTED DATA STORAGE SYSTEM COMMISSIONING AND 63/082,624 Sep. 24,2020 DECOMMISSIONING METADATA NODES IN A RUNNING DISTRIBUTED DATASTORAGE SYSTEM CONTAINER DATA MOVER FOR 63/082,631 Sep. 24, 2020MIGRATING DATA BETWEEN DISTINCT DISTRIBUTED DATA STORAGE SYSTEMSINTEGRATED WITH APPLICATION ORCHESTRATORS

Distributed Data Storage System

An example embodiment of the disclosed distributed data storage systemis the Hedvig Distributed Storage Platform now available from CommvaultSystems, Inc. of Tinton Falls, N.J., USA, and thus some of theterminology herein originated with the Hedvig product line.

The illustrative distributed data storage system comprises a pluralityof storage service nodes that form a storage cluster. Data reads andwrites originating from an application on an application host computingdevice are intercepted by a storage proxy, which is co-resident with theoriginating application. The storage proxy performs some pre-processingand analysis functions before making communicative contact with thestorage cluster. The system ensures strong consistency of data andmetadata written to the storage service nodes.

Terminology for the Distributed Data Storage System

Data and Metadata. To enhance the reader's understanding of the presentdisclosure, the term “metadata” is distinguished from the term “data”herein, even though both data and metadata comprise information storedon the illustrative distributed data storage system. Accordingly, “data”will refer to “payload” data, which is typically generated by anapplication or other data source that uses the distributed data storagesystem for data storage. Thus, the terms “data”, “payload”, and “payloaddata” will be used interchangeably herein. On the other hand, “metadata”will refer to other information in the distributed data storage system,e.g., information about the payload data, about the components hostingthe payload data, about metadata-hosting components, about othercomponents of the distributed data storage system, and also informationabout the metadata, i.e., “meta-metadata.”

Storage Service, e.g., Hedvig Storage Service. The storage service is asoftware component that installs on commodity x86 or ARM servers totransform existing server and storage assets into a fully-featuredelastic storage cluster. The storage service may deploy to anon-premises infrastructure, to hosted clouds, and/or to public cloudcomputing environments to create a single storage cluster.

Storage Service Node (or storage node), e.g., Hedvig Storage Server(HSS), comprises both computing and storage resources that collectivelyprovide storage service. The system's storage service nodes collectivelyform a storage cluster. One or more of the following storage servicesubsystems of the storage service may be instantiated at and may operateon a storage service node: (i) distributed fault-tolerant metadatasubsystem providing metadata service, e.g., “Hedvig Pages”; (ii)distributed fault-tolerant data subsystem (or data storage subsystem)providing payload data storage, e.g., “Hedvig HBlock”; and (iii)distributed fault-tolerant pod subsystem for generating and maintainingcertain system-level information, e.g., “Hedvig HPod.” The system storespayload data on certain dedicated storage resources managed by the datastorage subsystem, and stores metadata on other dedicated storageresources managed by the metadata subsystem. Thus, another way todistinguish payload data from metadata in the illustrative system isthat payload data is stored in and maintained by the data storagesubsystem and metadata is stored in and maintained by the metadatasubsystem. The pod subsystem, the metadata subsystem, and the datastorage subsystem are all partitioned and replicated across variousstorage service nodes. These subsystems operate as independent services,they need not be co-located on the same storage service node, and theymay communicate with a subsystem on another storage service node asneeded.

Replica. The distributed data storage system replicates data andmetadata across multiple storage service nodes. A “replica” or “replicanode” is a storage service node that hosts a replicated copy of dataand/or metadata that is also stored on other replica nodes.Illustratively, metadata uses a replication factor of 3, though theinvention is not so limited. Thus, with a replication factor of 3(“RF3”), each portion of metadata is replicated on three distinctmetadata nodes across the storage cluster.

Virtual Disk (“vdisk”) and Storage Containers. The virtual disk is theunit of storage made visible by system 100 to applications and/orapplication nodes. Every virtual disk provisioned on the system ispartitioned into fixed size chunks, each of which is called a storagecontainer. Different replicas are assigned for each storage container.Since replica assignment occurs at the storage container level—not at avirtual disk level—the data for a virtual disk is distributed across aplurality of storage service nodes, thus allowing increased parallelismduring I/Os and/or disk rebuilds. Thus, virtual disks are distributedand fault-tolerant.

Storage Pools. Storage pools are logical groupings of physicaldisks/drives in a storage service node and are configured as theprotection unit for disk/drive failures and rebuilds. Within a replica,one or more storage containers are assigned to a storage pool. A typicalstorage service node will host two to four storage pools.

Metadata Node. An instance of the metadata subsystem executing on astorage service node is referred to as a metadata node that provides“metadata service.” The metadata subsystem executing on a storageservice node stores metadata at the storage service node. The metadatanode communicates with one or more other metadata nodes to provide asystem-wide metadata service. The metadata subsystem also communicateswith pod and/or data storage subsystems at the same or other storageservice nodes. Some metadata nodes are designated owners of certainvirtual disks whereas others are replicas but not owners. Owner nodesare invested with certain functionality for managing the owned virtualdisk.

Metadata Node Identifier or Storage Identifier (SID) is a uniqueidentifier of the metadata service instance on a storage service node,i.e., the unique system-wide identifier of a metadata node.

Storage Proxy. Each storage proxy is a lightweight software componentthat deploys at the application tier, i.e., on application servers orhosts. A storage proxy may be implemented as a virtual machine (VM) oras a software container (e.g., Docker), or may run on bare metal toprovide storage access to any physical host or VM in the applicationtier. As noted, the storage proxy intercepts reads and writes issued byapplications and directs input/output (I/O) requests to the relevantstorage service nodes.

Erasure Coding (EC). In some embodiments, the illustrative distributeddata storage system employs erasure coding rather than or in addition toreplication. EC is one of the administrable attributes for a virtualdisk. The default EC policy is (4,2), but (8,2) and (8,4) are alsosupported if a sufficient number of storage service nodes are available.The invention is not limited to a particular EC policy unless otherwisenoted herein.

FIG. 1A is a block diagram depicting a distributed data storage system100 according to an illustrative embodiment. The figure depicts: aplurality of application nodes 102 that form an “application tier,” eachapplication node comprising a storage proxy 106 and one of components103A, 104A, and 105A; and a storage cluster 110 comprising a pluralityof separately scalable storage service nodes 120 and a plurality ofspecially-equipped compute hosts 121.

Distributed data storage system 100 (or system 100) comprises storageproxies 106 and storage cluster 110. System 100 flexibly leverages bothhyperscale and hyperconverged deployment options, sometimes implementedin the same storage cluster 110 as depicted here. Hyperscale deploymentsscale storage resources independently from the application tier, asshown by storage service nodes 120 (e.g., 120-1 . . . 120-N). In suchhyperscale deployments, storage capacity and performance scale outhorizontally by adding commodity servers running the illustrativestorage service; application nodes (or hosts) 102 scale separately alongwith storage proxy 106. On the other hand, hyperconverged deploymentsscale compute and storage in lockstep, with workloads and applicationsresiding on the same physical nodes as payload data, as shown by computehosts 121. In such hyperconverged deployments, storage proxy 106 andstorage service software 122 are packaged and deployed as VMs on acompute host 121 with a hypervisor 103 installed. In some embodiments,system 100 provides plug-ins for hypervisor and virtualization tools,such as VMware vCenter, to provide a single management interface for ahyperconverged solution.

System 100 provides enterprise-grade storage services, includingdeduplication, compression, snapshots, clones, replication,auto-tiering, multitenancy, and self-healing of both silent corruptionand/or disk/node failures to support production storage operations,enterprise service level agreements (SLAs), and/or robust storage forbacked up data (secondary copies). Thus, system 100 eliminates the needfor enterprises to deploy bolted-on or disparate solutions to deliver acomplete set of data services. This simplifies infrastructure andfurther reduces overall Information Technology (IT) capital expendituresand operating expenses. Enterprise storage capabilities can beconfigured at the granularity of a virtual disk, providing each dataoriginator, e.g., application, VM, and/or software container, with itsown unique storage policy. Every storage feature can be switched on oroff to fit the specific needs of any given workload. Thus, the granularprovisioning of features empowers administrators to avoid the challengesand compromises of “one size fits all” storage and helps effectivelysupport business SLAs, while decreasing operational costs.

System 100 inherently supports multi-site availability, which removesthe need for additional costly disaster recovery solutions. The systemprovides native high availability storage for applications acrossgeographically dispersed data centers by setting a unique replicationpolicy and replication factor at the virtual disk level.

System 100 comprises a “shared-nothing” distributed computingarchitecture in which each storage service node is independent andself-sufficient. Thus, system 100 eliminates any single point offailure, allows for self-healing, provides non-disruptive upgrades, andscales indefinitely by adding more storage service nodes. Each storageservice node stores and processes metadata and/or payload data, thencommunicates with other storage service nodes for data/metadatadistribution according to the replication factor.

Storage efficiency in the storage cluster is characterized by a numberof features, including: thin provisioning, deduplication, compression,compaction, and auto-tiering. Each virtual disk is thinly provisioned bydefault and does not consume capacity until data is written therein.This space-efficient dynamic storage allocation capability is especiallyuseful in DevOps environments that use Docker, OpenStack, and othercloud platforms where volumes do not support thin provisioninginherently, but can support it using the virtual disks of system 100.System 100 provides inline global deduplication that delivers spacesavings across the entire storage cluster. Deduplication isadministrable at the virtual disk level to optimize I/O and lower thecost of storing data. As writes occur, the system 100 calculates theunique fingerprint of data blocks and replaces redundant data with asmall pointer. The deduplication process can be configured to begin atstorage proxy 106, improving write performance and eliminating redundantdata transfers over the network. System 100 provides inline compressionadministrable at the virtual disk level to optimize capacity usage. Thesystem stores only compressed data on the storage service nodes.Illustratively, the Snappy compression library is used, but theinvention is not limited to this implementation. To improve readperformance and optimize storage space, the illustrative systemperiodically performs garbage collection to compact redundant blocks andgenerate large sequential chunks of data. The illustrative systembalances performance and cost by supporting tiering of data amonghigh-speed SSDs and lower-tier persistent storage technologies.

Application node (or host) 102 (e.g., 102-1, 102-2, 102-3) is anycomputing device, comprising one or more hardware processors andcomputer memory for executing computer programs, that generates and/oraccesses data stored in storage cluster 110. Application(s) (not shownhere but see, e.g., applications 132 in FIG. 1B) executing on anapplication node 102 use storage cluster 110 as a data storage resource.Application node 102 can take the form of: a bare metal host 105A forapplications with storage proxy 106-3; a virtual machine server withhypervisor 103A and storage proxy 106-1; a container host hostingsoftware container 104A and storage proxy 106-2; and/or anothercomputing device configuration equipped with a storage proxy 106.

Hypervisor 103 (e.g., 103A, 103B) is any hypervisor, virtual machinemonitor, or virtualizer that creates and runs virtual machines on avirtual machine server or host. Software container 104A is any operatingsystem virtualization software that shares the kernel of the hostcomputing device (e.g., 102, 121) that it runs on and allows multipleisolated user space instances to co-exist. Docker is an example ofsoftware container 104A. Bare metal 105A refers to application node102-3 running as a traditional computing device without virtualizationfeatures. Components 103, 104A, and 105A/B are well known in the art.

Storage proxy 106 (e.g., 106-1, 106-2, 106-3, 106-J . . . 106-K) is alightweight software component that deploys at the application tier,i.e., on application nodes 102 and/or compute hosts 121. A storage proxymay be implemented as a virtual machine 106-1, as a software container(e.g., Docker) 106-2, and/or running on bare metal (e.g., 106-3) toprovide storage access to any physical host or VM in the applicationtier. The storage proxy acts as a gatekeeper for all I/O requests tovirtual disks configured at storage cluster 110. It acts as a storageprotocol converter, load balances I/O requests to storage service nodes,caches data fingerprints, and performs certain deduplication functions.Storage protocols supported by storage proxy 106 include Internet SmallComputer Systems Interface (iSCSI), Network File System (NFS), ServerMessage Block (SMB2) or Common Internet File System (CIFS), AmazonSimple Storage Service (S3), OpenStack Object Store (Swift), withoutlimitation. The storage proxy runs in user space and can be managed byany virtualization management or orchestration tool. With storageproxies 106 that run in user space, the disclosed solution is compatiblewith any hypervisor, software container, operating system, or bare metalcomputing environment at the application node. In some virtualizedembodiments where storage proxy 106 is deployed on a virtual machine,the storage proxy may be referred to as a “controller virtual machine”(CVM) in contrast to application-hosting virtual machines that generatedata for and access data at the storage cluster.

Storage cluster 110 comprises the actual storage resources of system100, such as storage service nodes 120 and storage services 122 runningon compute hosts 121. In some embodiments, storage cluster 110 is saidto comprise compute hosts 121 and/or storage service nodes 120.

Storage service node 120 (e.g., 120-1 . . . 120-N) is any commodityserver configured with one or more x86 or ARM hardware processors andwith computer memory for executing the illustrative storage service,which is described in more detail in FIG. 1C. Storage service node 120also comprises storage resources as described in more detail in FIG. 1D.By running the storage service, the commodity server is transformed intoa full-featured component of storage cluster 110. System 100 maycomprise any number of storage service nodes 120.

Compute host 121 (e.g., 121-1 . . . 121-M) is any computing device,comprising one or more hardware processors and computer memory forexecuting computer programs, that comprises the functional components ofan application node 102 and of a storage service node 120 in a“hyperconverged” configuration. In some embodiments, compute hosts 121are configured, sometimes in a group, within an appliance such as theCommvault Hyperscale™ X backup appliance from Commvault Systems Inc., ofTinton Falls, N.J., USA.

FIG. 1B is a block diagram illustrating some details of distributed datastorage system 100 comprising separately scalable storage service nodes120 according to an illustrative embodiment. The figure depicts:application node 102-1 embodied as a VM host and hosting hypervisor 103,storage proxy 106-1 embodied as a controller virtual machine, and clientVM 131 hosting application 132-1; application node 102-2 hostingcontainerized storage proxy 106-2 and containerized application 132-2;and storage cluster 110 comprising nine (9) distinct physical storageservice nodes 120 (e.g., 120-1 . . . 120-9). Virtual machine hosts,virtual machines, and hypervisors are well known in the art.

Application 132 (e.g., 132-1, 132-2) is any software that executes onits underlying host (e.g., 102-1, 102-2) and performs a function as aresult. The application 132 may generate data and/or need to access datawhich is stored in system 100. Examples of application 132 include emailapplications, database management applications, office productivitysoftware, backup software, etc., without limitation.

The bi-directional arrows between each storage proxy 106 and a storageservice node 120 depict the fact that communications betweenapplications 132 and storage cluster 110 pass through storage proxies106, each of which identifies a proper storage service node 120 tocommunicate with for the present transaction, e.g., storage service node120-2 for storage proxy 106-1, storage service node 120-4 for storageproxy 106-2.

FIG. 1C is a block diagram depicting certain subsystems of the storageservice of distributed data storage system 100, according to anillustrative embodiment. Depicted here are: storage proxy 106;application 132; and a storage service node 120 comprising a podsubsystem 130 (e.g., Hedvig “HPOD”), a metadata subsystem 140 (e.g.,Hedvig “PAGES”), and a data storage subsystem 150 (e.g., Hedvig“HBLOCK”). Although storage service node 120 as depicted here comprisesan instance of all three storage service subsystems, any given storageservice node 120 need not comprise all three subsystems. Thus, asubsystem running on a given storage service node may communicate withone or more subsystems on another storage service node as needed tocomplete a task or workload.

Storage proxy 106 intercepts reads and writes issued by applications 132that are targeted to particular virtual disks configured in storagecluster 110. Storage proxy 106 provides native block, file, and objectstorage protocol support, as follows:

Block storage—system 100 presents a block-based virtual disk through astorage proxy 106 as a logical unit number (LUN). Access to the LUN,with the properties applied during virtual disk provisioning, such ascompression, deduplication and replication, is given to a host as aniSCSI target. After the virtual disk is in use, the storage proxytranslates and relays all LUN operations to the underlying storagecluster.

File storage—system 100 presents a file-based virtual disk to one ormore storage proxies 106 as an NFS export, which is then consumed by thehypervisor as an NFS datastore. Administrators can then provision VMs onthat NFS datastore. The storage proxy acts as an NFS server that trapsNFS requests and translates them into the appropriate remote procedurecall (RPC) calls to the backend storage service node.

Object storage—buckets created via the Amazon S3 API, or storagecontainers created via the OpenStack Swift API, are translated via thestorage proxies 106 and internally mapped to virtual disks 170 (shown inFIG. 1D). The storage cluster 110 acts as the object (S3/Swift) target,which client applications 132 can utilize to store and access objects.

Storage Proxy 106 comprises one or more caches that enable distributedoperations and the performing of storage system operations locally atthe application node 102 to accelerate read/write performance andefficiency. An illustrative metacache stores metadata locally at thestorage proxy, preferably on SSDs. This cache eliminates the need totraverse the network for metadata lookups, leading to substantial readacceleration. For virtual disks provisioned with client-side caching, anillustrative block cache stores data blocks to local SSD drives toaccelerate reads. By returning blocks directly from the storage proxy,read operations avoid network hops when accessing recently used data.For virtual disks provisioned with deduplication, an illustrative dedupecache resides on local SSD media and stores fingerprint information ofcertain data blocks written to storage cluster 110. Based on this cache,the storage proxy determines whether data blocks have been previouslywritten and if so, avoids re-writing these data blocks again. Storageproxy 106 first queries the dedupe cache and if the data block is aduplicate, storage proxy 106 updates metadata subsystem 140 to map thenew data block(s) and acknowledges the write to originating application132. Otherwise, storage proxy 106 queries metadata subsystem 140 and ifthe data block was previously written to storage cluster 110, the dedupecache and metadata subsystem 140 are updated accordingly, with anacknowledgement to originating application 132. Unique new data blocksare written to the storage cluster as new payload data. More details onreads and writes are given in FIGS. 1E and 1F.

A simplified use case workflow comprises: 1. A virtual disk 170 isadministered with storage policies via a web-based user interface, acommand line interface, and/or a RESTful API (representational statetransfer application programming interface). 2. Block and file virtualdisks are attached to a storage proxy 106, which presents the storageresource to application hosts, e.g., 102. For object storage,applications 132 directly interact with the virtual disk via Amazon S3or OpenStack Swift protocols. 3. Storage proxy 106 interceptsapplication 132 I/O through the native storage protocol and communicatesit to the underlying storage cluster 110 via remote procedure calls(RPCs). 4. The storage service distributes and replicates datathroughout the storage cluster based on virtual disk policies. 5. Thestorage service conducts background processes to auto-tier and balanceacross racks, data centers, and/or public clouds based on virtual diskpolicies.

Pod subsystem 130 maintains certain system-wide information forsynchronization purposes and comprises processing and tracking resourcesand locally stored information. A network of pods 130 throughout storagecluster 110, where each pod comprises three nodes, is used for managingtransactions for metadata updates, distributed-atomic-counters as aservice, tracking system-wide timeframes such as generations and epochs,etc. More details on the pod subsystem may be found in U.S. Pat. No.9,483,205 B2, which is incorporated by reference in its entirety herein.

Metadata subsystem 140 comprises metadata processing resources andpartitioned replicated metadata stored locally at the storage servicenode. Metadata subsystem 140 receives, processes, and generatesmetadata. Metadata in system 100 is partitioned and replicated across aplurality of metadata nodes. Typically, metadata subsystem 140 isconfigured with a replication factor of 3 (RF3), and therefore many ofthe examples herein will include 3-way replication scenarios, but theinvention is not so limited. Each metadata subsystem 140 tracks thestate of data storage subsystems 150 and of other metadata subsystems140 in storage cluster 110 to form a global view of the cluster.Metadata subsystem 140 is responsible for optimal replica assignment andtracks writes in storage cluster 110.

Metadata synchronization logic (or “anti-entropy engine” (AE) not shownhere) runs in metadata subsystem 140. The metadata synchronization logiccompares replicas of metadata across metadata nodes and ensures that thereplicas agree on a superset of the metadata therein to avoid losingmetadata. During storage and compaction of metadata-carryingstring-sorted tables (SSTs), a consistent file identification scheme isused across all metadata nodes. When an application node writes to andreads from a virtual disk on distributed data storage system 100,metadata is generated and stored in replicas on different metadatanodes. A modified log-structured merge tree is used to store and compactthe metadata SST files. A fingerprint file is created for each metadataSST file that includes a start-length-hash value triple for each regionof the metadata SST file. To synchronize, fingerprint files of twometadata SST files are compared, and if any hash values are missing froma fingerprint file, then key-value-timestamp triples corresponding tothese missing hash values are sent to the metadata SST file that ismissing them. An example of metadata synchronization logic is describedin U.S. Pat. No. 10,740,300, which is incorporated by reference in itsentirety herein.

Data storage subsystem 150 receives, processes, and stores payload datawritten to storage cluster 110. Thus, data storage subsystem 150 isresponsible for replicating data to other data storage subsystems 150 onother storage service nodes and striping data within and across storagepools. Data storage subsystem 150 comprises storage processing forpayload data blocks (e.g., I/O, compaction, garbage collection, etc.)and stores partitioned replicated payload data at the storage servicenode.

The bold bi-directional arrows in the present figure show that metadatais communicated between storage proxy 106 and metadata subsystem 140,whereas data blocks are transmitted to/from data storage subsystem 150.Depending on the configuration, metadata subsystem 140 may operate on afirst storage service node 120 or storage service 122 and data storagesubsystem 150 may operate on another distinct storage service node 120or storage service 122. See also FIGS. 1E and 1F.

FIG. 1D is a block diagram depicting a virtual disk distributed across aplurality of storage service nodes and also depicting a plurality ofstorage resources available at each storage service node according to anillustrative embodiment. The present figure depicts: nine storageservice nodes 120 (120-1 . . . 120-09); a virtual disk 170 thatcomprises data distributed over four of the storage service nodes—120-1,120-2, 120-4, and 120-5; and storage resources 160 configured withinstorage service node 120-9.

Each storage service node 120 (or compute host 121) is typicallyconfigured with computing resources (e.g., hardware processors andcomputer memory) for providing storage services and with a number ofstorage resources 160, e.g., hard disk drives (HDD) shown here asstorage disk shapes, solid state storage drives (SSD) (e.g., flashmemory technology) shown here as square shapes, etc. The illustrativesystem uses commit logs, which are preferably stored on SSD before theyare flushed to another disk/drive for persistent storage. Metadatacommit logs are stored on dedicated metadata-commit-log drives “MCL”,whereas payload-data commit logs are stored on distinct dedicateddata-commit-log drives “DCL.” As an example depicted in the presentfigure, pod system information is stored in storage resource “P” whichis preferably SSD technology for faster read/write performance; themetadata commit log is stored in storage resource “MCL” which ispreferably SSD technology; metadata is then flushed from the commit logto persistent storage “M” (SSD and/or HDD); the data commit log isstored in storage resource “DCL” which is preferably SSD technology;payload data is then flushed from the data commit log to persistentstorage “D” (typically HDD). The storage resources 160 depicted in thepresent figures are shown here as non-limiting examples to ease thereader's understanding; the numbers and types of storage technologiesamong storage resources 160 will vary according to differentimplementations.

To accelerate read operations, client-side caching of data is used onSSDs accessible by storage proxy 106. Data is also cached on SSDs atstorage service nodes. For caching, the system supports the use ofPeripheral Component Interconnect Express (PCIe) and Non-Volatile MemoryExpress (NVMe) SSDs. All writes are executed in memory and flash(SSD/NVMe) and flushed sequentially to persistent storage. Persistentstorage uses flash technology (e.g., multi-level cell (MLC) and/or 3DNAND SSD) and/or spinning disk technology (e.g., HDD)). Options areadministrable at the virtual disk level.

Virtual disk (“vdisk”) 170 is the data storage representation of system100 that is visible to and accessible by applications 132 as datastorage resources. Virtual disk 170 is also referred to herein as “uservirtual disk” 170 to reflect that it is visible to applications 132 as adata storage resource. In other words, each application 132 will use oneor more virtual disks 170 for data storage without having knowledge ofhow system 100 as a whole is organized and configured. Every virtualdisk 170 provisioned on the system is partitioned into fixed sizechunks, each of which is called a storage container. Different replicasare assigned for each storage container. Since replica assignment occursat the storage container level—not at a virtual disk level—the data fora virtual disk is distributed across a plurality of storage servicenodes, thus allowing increased parallelism during I/Os and/or diskrebuilds. Thus, the virtual disks are distributed and fault-tolerant.Notably, the replication factor alone (e.g., RF3) does not limit howmany storage service nodes 120 may comprise payload data of a givenvirtual disk 170. Thus, different containers of the virtual disk may bestored and replicated on different storage service nodes, adding up tomore total storage service nodes associated with the virtual disk thanthe replication factor of the virtual disk.

Any number of virtual disks 170 may be spun up, each one thinlyprovisioned and instantly available. Illustrative user-configurableattributes for virtual disk 170 include without limitation: Name—aunique name to identify the virtual disk. Size—to set the desiredvirtual disk size. System 100 supports single block and NFS virtualdisks of unlimited size. Disk Type—to specify the type of storageprotocol to use for the virtual disk: block or file (NFS). Objectcontainers/buckets are provisioned directly from OpenStack via Swift,via the Amazon S3 API, etc. Workload Type—for NFS disk type, optionsinclude default, proprietary, or object storage target (OST) workloadtypes. For proprietary and OST, if Enable Deduplication is selected, aRetention Policy can be added as well. For block disk type, the onlyoption is default. Retention Policy—specifies a duration for proprietaryand OST workloads, e.g., two weeks, one month, etc. Encryption—toencrypt both data at rest and data in flight for the virtual disk.Enable Deduplication—to enable inline global deduplication. ClusteredFile System—to indicate that the virtual disk will be used with aclustered file system. When selected, system 100 enables concurrentread/write operations from multiple VMs or hosts. Description—to providean optional brief description of the virtual disk. Compressed—to enablevirtual disk compression to reduce data size. Client-Side Caching—tocache data to local SSD or PCIe devices at the application tier toaccelerate read performance. CSV—to enable Cluster Shared Volumes forfailover (or high availability) clustering. A CSV is a shared diskcontaining a Windows NT File System (NTFS) or Resilient File System(ReFS) volume that is made accessible for read and write operations byall nodes within a Windows Server failover cluster. ReplicationPolicy—to set the policy for how data will replicate across the cluster:Agnostic, Rack Aware, or Data Center Aware. Replication Factor (RF)—todesignate the number of replicas for each virtual disk. Replicationfactor is tunable, typically ranging from one to six, withoutlimitation. Block Size—to set a block virtual disk size to 512 bytes, 4kor 64k. File (NFS)-based virtual disks have a standard 512 size, andobject-based virtual disks have a standard 64K size. Residence—to selectthe type of media on which the data is to reside: HDD, SSD. The presentfigure depicts only one virtual disk 170 for illustrative purposes, butsystem 100 has no limits on how many virtual disks it may support.

FIG. 1E is a block diagram depicting a typical I/O workflow for writeoperations originating with an application. This figure depicts anapplication 132 writing to storage cluster 110, illustratively writingto a virtual disk 170 configured with Replication Factor=3 (RF3).

At step W, storage proxy 106 intercepts a write command issued byapplication 132, comprising one or more payload data blocks to bewritten to a virtual disk 170 in storage cluster 110. At step 1W,storage proxy 106 determines the replica nodes 120 for the data blocksto be written and transmits the data blocks to one of the replica nodes120, e.g., 120-4. If the virtual disk is enabled for deduplication,storage proxy 106 calculates a data block fingerprint, queries thededupe cache and, if necessary, further queries metadata subsystem 140(at the virtual disk's metadata owner node, e.g., 120-7), and eithermakes a metadata update or proceeds with a new write. At step 2W, datastorage subsystem 150 on replica node 120-4 receives and writes the datablocks locally and forwards them to other designated replica nodes,e.g., 120-1 and 120-8. At step 3W, storage proxy 106 sends a writeacknowledgment back to the originating application 132 after a quorum ofdata storage subsystem 150 replicas have completed step 2W. For RF3, twoacknowledged successful writes are needed from the three (RF3) replicasto satisfy the quorum (RF/2+1=3/2+1=2). Two of the three replicas arewritten synchronously, and one may be written asynchronously. At step4W, storage proxy 106 causes an atomic write to be made into metadatasubsystem 140 at metadata owner node 120-7, after which the write isdeemed successful. At step 5W, metadata subsystem 140 replicates themetadata from node 120-7 to designated metadata replica nodes, e.g.,120-8 and 120-9.

FIG. 1F is a block diagram depicting a typical I/O workflow for readoperations originating with an application. This figure depicts anapplication 132 reading from storage cluster 110, illustratively readingfrom a virtual disk 170 configured with RF3.

At step R, storage proxy 106 intercepts a read request issued byapplication 132 for one or more data blocks from a virtual disk 170 instorage cluster 110. At step 1R, storage proxy 106 queries the localmetacache for a particular data block to be read and if the informationis not found in the local metacache, at step 1R′ storage proxy 106consults metadata subsystem 140 (e.g., at the vdisk's designatedmetadata owner node 120-7). At step 2R, storage proxy 106 sends the datablock details to one of the closest data storage subsystems 150, basedon observed latency, e.g., storage service node 120-4. At step 3R, datastorage subsystem 150 reads the data block(s) and transmits the block(s)back, if found, to storage proxy 106. If the read operation fails due toany error, the read is attempted from another replica. At step 4R,storage proxy 106 serves the requested data block(s) to application 132.If client-side caching is enabled for the targeted virtual disk 170during provisioning, storage proxy 106 queries the local block cache atstep 1R to fetch the data block(s), and if found therein serves the datablock(s) to application 132 at step 4R, thereby bypassing data storagesubsystem 150 at the storage service nodes(s) and eliminating the needto traverse the network to reach storage cluster 110.

System Resiliency. System 100 is designed to survive disk, node, rack,and data center outages without application downtime and with minimalperformance impact. These resiliency features include: highavailability, non-disruptive upgrades (NDU), disk failures, replication,and snapshots and clones.

High availability (HA). A preferable minimum of three storage servicenode should be provisioned for an implementation of the illustrativesystem. Redundancy can be set as agnostic, at the rack level, or at datacenter level. The system initiates transparent failover in case offailure. During node, rack, or site failures, reads and writes continueas usual from/to remaining operational replicas. To protect against asingle point of failure, storage proxies 106 install as a highavailability active/passive pair (“HA pair,” not shown). A virtual IPaddress (VIP) assigned to the HA pair redirects traffic automatically tothe active storage proxy 106 at any given time. If one storage proxy 106instance is lost or interrupted, operations fail over seamlessly to thepassive instance to maintain availability. This happens withoutrequiring intervention by applications, administrators, or users. Duringprovisioning, administrators can indicate that an application host102/121 will use a clustered file system. This automatically setsinternal configuration parameters to ensure seamless failover when usingVM migration to a secondary physical host running its own storage proxy106. During live VM migration, such as VMware vMotion or MicrosoftHyper-V, any necessary block and file storage “follows” guest VMs toanother host.

Non-disruptive upgrades (NDUs). The illustrative system supportsnon-disruptive software upgrades by staging and rolling the upgradeacross individual components using the highly available nature ofdistributed data storage system 100 to eliminate any downtime or dataunavailability. Storage service nodes 120 and storage services 122undergo upgrades first one node at a time. Meanwhile, any I/O continuesto be serviced from alternate available nodes, e.g., replicas. Storageproxies 106 are upgraded next, starting with the passive storage proxyin HA pairs. After the passive storage proxy upgrade is complete, it ismade active, and the formerly active storage proxy 106 is upgraded andresumes service as the passive of the HA pair. This process eliminatesany interruption to reads or writes during the upgrade procedure.

Disk Failures. The illustrative system supports efficient data andmetadata rebuilds that are initiated automatically when there is a diskfailure. Payload data is rebuilt from other data replicas and usinginformation in the metadata subsystem. The metadata rebuild self-healswithin the metadata service.

Replication. The illustrative system uses a combination of synchronousand asynchronous replication processes to distribute and protect dataacross the cluster and provide near-zero recovery point objectives (RPO)and recovery time objectives (RTO). For example, two of three replicasare written synchronously, and one is written asynchronously. The systemsupports any number of active data centers in a single storage cluster110, using a tunable replication factor and replication policy options.The replication factor designates the number of replicas to create foreach virtual disk, and the replication policy defines the destinationfor the replicas across the cluster. Replicas occur at the storagecontainer level of a virtual disk 170. For example, if a 100 GB virtualdisk with RF3 is created, the entire 100 GBs are not stored ascontiguous chunks on three storage service nodes. Instead, the 100 GBsare divided among several storage containers, and replicas of eachstorage container are spread across different storage pools on differentstorage service nodes within the storage cluster. For additionaldisaster recovery protection against rack and data center failures, theillustrative system supports replication policies that span multipleracks or data centers using structured IP addressing, DNS naming/suffix,and/or customer-defined snitch endpoints. For “agnostic” replicationpolicies, data is spread across the storage cluster using a best-effortto improve availability. For “rack aware” replication policies, data isspread across as many physically distinct racks as possible within in asingle data center. For “data center aware” replication policies, datareplicates to additional physical sites, which can include privateand/or hosted data centers and public clouds. In a disaster recoveryexample, where the Replication Policy=Data Center Aware and theReplication Factor=3, the illustrative system divides the data intostorage containers and ensures that three copies (RF3) of each storagecontainer are spread to geographically dispersed physical sites, e.g.,Data Centers A, B, and C. At any time, if a data copy fails,re-replication is automatically initiated from replicas across the datacenters.

Snapshots And Clones. In addition to replication policies, datamanagement tasks include taking snapshots and making “zero-copy” clonesof virtual disks. There is no limit to the number of snapshots or clonesthat can be created. Snapshots and clones are space-efficient, requiringcapacity only for changed blocks.

Encryption. The illustrative system provides software-based encryptionwith the Encrypt360 feature. This enables encryption of data at thepoint of ingestion (at storage proxy 106). Data encrypted in this wayremains protected in flight between storage proxy 106 and storageservice nodes 120/storage service 122, in flight among storage servicenodes as part of replication, in-use at storage proxy 106, and at restwhile in storage. Any encryption scheme may be implemented, preferably256-bit AES. Additionally, any third-party key management system can beattached.

Ecosystem Integration. The illustrative system works with and provides asecure distributed data storage system for a variety of data-generatingplatforms, including systems that generate primary (production) data andsystems that generate backup data from primary sources. VMware. Theillustrative system features a vCenter plug-in that enablesprovisioning, management, snapshotting, and cloning of virtual disks 170directly from the vSphere Web Client. Additionally, the systemincorporates support for the VMware vSphere Storage APIs ArrayIntegration (VAAI). Docker. The illustrative system provides persistentstorage for Docker software containers through a volume plugin. Thevolume plugin enables a user to create a persistent Docker volume backedby a virtual disk 170. Different options, such as deduplication,compression, replication factor, and/or block size, may be set for eachDocker volume, using “volume options” in the Docker Universal ControlPlane (UCP) or using the “docker volume” command line. The virtual diskcan then be attached to any host. The volume plugin also creates a filesystem on this virtual disk and mounts it using the path provided by theuser. The file system type can also be configured by the user. All I/Oto the Docker volume goes to virtual disk 170. As the software containermoves in the environment, virtual disk 170 will automatically be madeavailable to any host, and data will be persisted using the policieschosen during volume creation. For container orchestration platforms,such as Kubernetes and OpenShift, the illustrative system 100 providespersistent storage for software containers through a proprietary dynamicprovisioner and via other technologies that interoperate with theorchestration platform(s). OpenStack. The illustrative system deliversblock, file, and object storage for OpenStack all from a single platformvia native Cinder and Swift integration. The system supports granularadministration, per-volume (Cinder) or per-container (Swift), forcapabilities such as compression, deduplication, snapshots, and/orclones. OpenStack administrators can provision the full set of storagecapabilities of system 100 in OpenStack Horizon via OpenStack's QoSfunctionality. As with VMware, administrators need not use system 100'snative web user interfaces and/or RESTful API, and storage can bemanaged from within the OpenStack interface.

Multitenancy. The illustrative system supports the use of rack-aware anddata center-aware replication policies for customers who must satisfyregulatory compliance and restrict certain data by region or site. Thesecapabilities provide the backbone of a multitenant architecture, whichis supported with three forms of architectural isolation: LUN masking,dedicated storage proxies, and complete physical isolation. Using theLUN masking option, different tenants are hosted on a sharedinfrastructure with logical separation. Logical separation is achievedby presenting virtual disks only to a certain VM and/or physicalapplication host (IP range). Quality of Service (QoS) is delivered atthe VM level. Using the dedicated storage proxies option, storage accessis provided with a dedicated storage proxy 106 per tenant. Storageproxies can be deployed on a dedicated physical host or a shared host.This provides storage as a shared infrastructure, while compute isdedicated to each tenant. Quality of Service (QoS) is at the VM level.Using the complete physical isolation option, different tenants arehosted on dedicated storage clusters (each running their own storageservice and storage proxies) to provide complete logical and physicalseparation between tenants. For all of these multitenant architectures,each tenant can have unique virtual disks with tenant-specific storagepolicies, because the illustrative system configures policies at thevirtual disk level. Policies can be grouped to create classes of service(CoS).

Thus, the illustrative distributed data storage system 100 scalesseamlessly and linearly from a few nodes to thousands of nodes usingvirtual disks as the user-visible storage resource provided by thesystem. Enterprise storage capabilities are configurable at the virtualdisk level. The storage service nodes can be configured in a pluralityof physical computing environments, e.g., data centers, private clouds,and/or public clouds without limitation. Likewise, the storage proxiesmay execute in the same or different computing environment from thestorage service nodes, e.g., within the same cloud computingenvironment, different cloud computing environments, different cloudavailability zones, and/or in a non-cloud data center, thus enablingcloud-to-cloud and/or multi-cloud services, as well as non-cloud and/orhybrid service environments.

Optimized Deduplication Based on Backup Frequency in a Distributed DataStorage System

FIG. 1G is a block diagram illustrating a backup configuration thatdepicts an illustrative data storage management system using distributeddata storage system 100 for storing secondary (backup) copies accordingto an illustrative embodiment. The figure depicts: storage service nodes120 configured with a global system deduplication virtual disk 1700;client computing device 1020 hosting application(s) 132 and dataagent(s) 1420; backup computing device 1060 hosting media agent(s) 1440and storage proxy 106; and storage manager 1400. An example of theillustrative data storage management system is the Commvault Complete™Backup and Recovery software solution from Commvault Systems, Inc. ofTinton Falls, N.J., USA. Hereinafter, secondary copies, which aredistinguishable from primary (application-native) data, will be referredto as “backup copies” for simplicity and to ease the reader'sunderstanding of the present disclosure.

Client computing device 1020 and backup computing device 1060 eachcomprise one or more hardware processors and computer memory forexecuting computer programs. Likewise, storage manager 1400 is hosted byand/or comprises one or more hardware processors and computer memory forexecuting computer programs. These components may operate in anycomputing environment, e.g., non-cloud data center, hybrid cloud,private cloud, and/or public cloud without limitation.

Components of the data storage management system include storage manager1400, one or more data agents 1420, and one or more media agents 1440.Primary data 1120 generated and used by client applications 132 iscaptured by a data agent 1420, transmitted to a media agent 1440, andconverted into one or more backup copies that are sent to distributeddata storage system 100 (via storage proxy 106) for storage. Control andmanagement of the backup process is performed by storage manager 1400.

Storage manager 1400 is a centralized storage and/or information managerthat is configured to perform certain control functions and also tostore certain critical information about the data storage managementsystem—hence storage manager 1400 is said to manage the data storagemanagement system. Storage manager 1400 communicates with, instructs,and/or controls data agents 1420 and media agents 1440. According tocertain embodiments, storage manager 1400 provides one or more of thefollowing functions:

-   -   communicating with data agents 1420 and media agents 1440,        including transmitting instructions, messages, and/or queries,        as well as receiving status reports, index information,        messages, and/or queries, and responding to same;    -   initiating execution of storage and/or information management        operations;    -   initiating restore and recovery operations;    -   allocating secondary storage resources for secondary copy        operations, e.g., distributed data storage system 100;    -   reporting, searching, and/or classification of data;    -   monitoring completion of and status reporting related to storage        operations, information management operations, and jobs;    -   tracking age information relating to backup copies and        initiating data pruning when appropriate;    -   protecting metadata of the data storage management system;    -   implementing job management, schedule management, event        management, alert management, reporting, job history        maintenance, user security management, disaster recovery        management, and/or user interfacing for system administrators        and/or end users of the data storage management system; etc.

Data agent 1420 is a component of the data storage management system andis generally directed by storage manager 1400 to participate in creatingor restoring backup copies. A variety of different applications 132 canoperate on a given client computing device 1020, including operatingsystems, file systems, database applications, e-mail applications, andvirtual machines, just to name a few. And, as part of the process ofcreating and restoring backup copies, the client computing device 1020may be tasked with processing and preparing the primary data 1120generated by these various applications 132. Moreover, the nature of theprocessing/preparation can differ among application types, e.g., due toinherent structural, state, and formatting differences amongapplications 132 and/or the operating system of client computing device1020. Each data agent 1420 is therefore advantageously configured toparticipate in storage operations and/or information managementoperations based on the type of primary data 1120 that is beingprotected at a client-specific and/or application-specific level. Dataagent 1420 may be a computer software program (e.g., in the form of aset of executable binary files) that executes on the same clientcomputing device 1020 as the associated application 132 that data agent1420 is configured to protect or on an associated computing device. Forinstance, data agent 1420 may take part in copying, archiving,migrating, and/or replicating of certain primary data 1120. Data agent1420 may receive control information from storage manager 1400, such ascommands to transfer copies of data objects and/or metadata to one ormore media agents 1440. Data agent 1420 also may format, compress,deduplicate, and/or encrypt certain primary data 1120, as well ascapture application-related metadata before transmitting the processeddata to media agent 1440. Data agent 1420 also may receive instructionsfrom storage manager 1400 to restore (or assist in restoring) a backupcopy such that the restored data may be properly accessed by application132 as primary data 1120 in an application-native format. Each dataagent 1420 may be specialized for a particular application 132.

Media agent 1440 is a component of the data storage management systemand is generally directed by storage manager 1400 in creating andrestoring backup copies such as backup copies stored at distributed datastorage system 100. Whereas storage manager 1400 generally manages thedata storage management system as a whole, media agent 1440 provides aportal to certain secondary storage resources, such as distributed datastorage system 100 by having specialized features for communicatingtherewith, e.g., via storage proxy 106. Media agent 1440 may be asoftware program (e.g., in the form of a set of executable binary files)that executes on a backup computing device 1060. Media agent 1440generally manages, coordinates, and facilitates the transmission of databetween a data agent 1420 and secondary storage resources (e.g., system100) associated with media agent 1440. For instance, other components inthe system may interact with media agent 1440 to gain access to datastored on distributed data storage system 100, (e.g., to browse, read,write, modify, delete, or restore data).

The configuration depicted in the present figure uses distributed datastorage system 100 as the storage target for backup copies. Backupcopies generated by the data storage management system are transmittedby each media agent 1440 to a user virtual disk 170 (not shown here, butsee virtual disk 170 in FIG. 1D), which is defined on the distributedstorage system as the storage target for the backup copy. Hereinafter,virtual disks 170 will be referred to as “user virtual disks” 170 todistinguish them more clearly from the global deduplication virtual disk1700. Storage proxy 106 intercepts the write requests issued by themedia agent 1440 and applies deduplication to the incoming data blocksas described herein.

The storage service nodes 120 comprise physical data storage resourcesas shown in another figure. The illustrative global system deduplicationvirtual disk 1700 is configured as a virtual disk that is partitionedand replicated across a plurality of storage service nodes 120/122.Thus, the global system deduplication virtual disk 1700 is treated as asingle logically-centralized repository of deduplicated data blocksacross the distributed storage system, but physical storage is bothpartitioned and replicated across a plurality of storage service nodes120/122.

The global system deduplication virtual disk (or “system-widededuplication virtual disk”) 1700 is partitioned into fixed size virtualchunks, each of which is called a storage container (illustrativelyembodied as a Hedvig Container). Different replicas are assigned foreach storage container. Since replica assignment occurs at the storagecontainer level, the data for a virtual disk such as the global systemdeduplication virtual disk 1700 is spread across the storage cluster.Replicas are chosen by metadata subsystem 140 according to replicationfactor and replication policy settings to support the application's dataprotection needs. See also FIG. 2B.

FIG. 1H is a block diagram illustrating a backup configuration in whicha third-party backup system uses the illustrative distributed datastorage system 100 for storing secondary (backup) copies according to anillustrative embodiment. This figure is analogous to FIG. 1G, exceptthat backup copies originate with a third-party backup system 1500rather than from the illustrative data storage management system of FIG.1G. Accordingly, backup data is generated by backup system 1500 andaddressed to a user virtual disk 170, which is defined on thedistributed storage system as the storage target for the backup copy.Storage proxy 106 intercepts the write requests and appliesdeduplication to the incoming data blocks as described herein.

FIG. 2A is a block diagram depicting components that play a role in thepresent deduplication solution according to an illustrative embodiment.The present figure depicts: storage proxy 106 comprising deduplicationtracking logic 206; pod subsystem 130 comprising epoch calculator 230;metadata subsystem 140 comprising deduplication tracking and garbagecollection logic 240; and data storage subsystem comprising compactionlogic 250 and write logic 252. Notably, the pod, metadata, and datastorage subsystems components need not reside on the same storageservice node, and may be deployed on various different storage servicenodes. More details are given in FIG. 2B.

FIG. 2B is a block diagram depicting certain components that play a rolein the present deduplication solution, providing additional detailsabout the components depicted in FIG. 2A.

Storage proxy 106 comprises deduplication management and tracking logic(e.g., “DD tracking logic”) 206 and an index (e.g., “DDCache”) 207 fortracking hash values and DDblockIDs with their associated expiry epochs.See also FIG. 2C. The DDCache 207 is where the DD tracking logic 206checks whether a hash value computed for an incoming data block can befound, and if so, identifies a unique DDblockID and an expiry epoch forit. DD tracking logic 206 at storage proxy 106 illustratively performsthe functionality of the storage proxy as depicted in FIG. 3.

Pod subsystem 130 uses an illustrative epoch calculator function 230 toprovide a cluster-level epoch value 231 used for the illustrativediscard or GC cycle. The current epoch value 231 is referred to hereinas Enow. In every discard cycle, Enow is incremented during the discardpreparation stage (+2) and incremented again during every discard stage(+1). Accordingly, the numerical values of Enow use the followingpattern: Enow=0, 2(+2 GC Prep), 3(+1 GC), 5(+2 GC Prep), 6(+1 GC), 8, 9,etc. The scheme for setting the current epoch, Enow, is also depicted inblock 401 of FIGS. 4 and 6.

Metadata subsystem 140 comprises deduplication management, tracking, andGC logic (e.g., the “DD tracking and garbage collection logic”) 240.Metadata subsystem 140 also maintains several data structures,illustratively organized as column families, e.g., VdiskBlockInfo 241,DDInfo 242, DDTracker 243, DDRefCount 244, GCCycleInfo 245, andContainer-Specific Discard Lists 246. See also FIGS. 2C-2G. DD trackingand garbage collection logic 240 illustratively performs thefunctionality of metadata subsystem 140 as depicted in FIG. 3, as wellas significant portions of methods 400 and 600 as depicted in FIGS. 4and 6, respectively.

Data storage subsystem 150 comprises write logic 252 for adding newDDblockIDs to the global system deduplication virtual disk 1700, or morespecifically, for writing new data blocks to the storage container(s)260 hosted by the particular storage service node, e.g., C1, C2, C3,which are numbered 260-1, 260-2, and 260-3, respectively. See also FIG.3. Data storage subsystem 150 receives container-specific discard lists246 from metadata subsystem 140. Data storage subsystem 150 alsocomprises compaction logic 250 that removes the DDblockIDs received inthe discard lists from the storage containers 260 at the storage servicenode 120/122. See also FIG. 6. Thus, each storage service node 120/122that hosts one or more storage containers 260 belonging to the globalsystem deduplication virtual disk 1700 is responsible for adding newblocks to and deleting blocks from those storage containers.

FIG. 2C depicts an illustrative column family for associating hashvalues with corresponding DDblockID and an associated expiry epoch,e.g., DDCache 207, DDInfo 242. This functions as an index of hashvalues. The DDCache 207 configured at a storage proxy 106 is configuredaccording to this illustrative column family. Likewise, the DDInfo 242configured at metadata subsystem 140 is also configured according tothis column family. Information from DDInfo 242 is sometimes updatedinto DDCache 207 as needed. See also FIG. 3. Expiry extensions forcertain DDblockIDs are updated into DDInfo 242 as needed during the GCphase of the discard cycle. See also FIG. 7.

FIG. 2D depicts an illustrative column family for tracking writerequests received by storage proxies, e.g., DDTracker 243. In eachdiscard cycle, a new DDTracker 243 column family tracks, for each uservirtual disk 170, every new write request intercepted by storage proxies106. The data is organized by storage container 260 belonging to theglobal system deduplication virtual disk 1700. For each DDblockID, itscorresponding hash value is included, and the DDblockID receives atimestamp of Enow+1. Notably, the DDTracker 243 tables are discardedafter their information is scanned and used for updating the DDRefCount244 table family. See also FIG. 4. Thus, DDTracker 243 providesinformation on write requests that came in during a certain discardcycle and DDTracker 243 is re-populated in the next discard cycle.

FIG. 2E depicts an illustrative column family that is updated duringeach discard preparation stage based on scanning and processingDDTracker 243 information, e.g., DDRefCount 244. The information isorganized according to storage containers 260 of the global systemdeduplication virtual disk 1700, and then by DDblockID. Columns for eachuser virtual disk 170 referencing the DDblockID are added. For everyundeleted DDblockID, the preparation stage processes information inDDTracker 243, updates DDRefCount 244 entries, and updates the DDblockIDexpiry column. The reference write epoch (Ref W Epoch) columns arepopulated from the timestamp column in DDTracker 243. The referenceexpiry value (Ref Expiry) columns add to the write epoch a frequency forexecuting full backups of the particular user virtual disk 170 thatreferences the DDblockID.

The expiry value assigned to the DDblockID (Eu) is the maximum value ofthe various Ref Expiry columns. Thus, the various backup frequencies ofthe various user virtual disks 170 are taken into consideration here.The Eu value is considered when deciding whether to discard a DDblockID.See also block 706 in FIG. 7. The Eu value is updated into DDInfo 242,if necessary, to extend the life of a DDblockID. See also block 709 inFIG. 7. The DDRefCount 244 column families are persisted across discardcycles, unlike DDTracker 243.

FIG. 2F depicts an illustrative column family that assigns the workloadof scanning DDTracker 243 column families (each DDTracker 243 associatedwith a specific user virtual disk 170) for the purpose of updatingDDRefCount 244. Various storage service nodes 120/122 are assigned thetask. This table is generated on a first-time execution of the discardpreparation stage and is persisted for future and repeated use in otherdiscard cycles. See also block 406 in FIG. 4.

FIG. 2G depicts an illustrative column family that comprises DDblockIDdiscard lists 246. These discard lists 246 are generated from theDDRefCount 244 column families during the discard (GC) stage, whichfollows the discard preparation stage. For each storage container 260belonging to the global system deduplication virtual disk 1700, discardlists 246 are added, one discard list 246 per epoch when the discardlist 246 was generated. The reason for multiple epochs showing up hereis that the compaction logic or process 250 that actually discardsDDblockIDs occurs asynchronously from any particular discard cycle, soit is possible for multiple lists 246 to accumulate before thecompaction process 250 is triggered on any given storage service node.

In regard to the column families depicted in FIGS. 2C-2G, the depictionsare illustrative and the invention is not so limited. In otherembodiments, the data may be differently organized and the variousstages responsible for generating and processing the data may alsodiffer from what is depicted and described herein.

FIG. 3 depicts a fence diagram that illustrates some salient operationsoccurring during an I/O cycle of an incoming data block intercepted by astorage proxy 106. The fence diagram depicts operations at storage proxy106 in the left-hand column (e.g., using DD tracking logic 206 andDDCache 207), operations at a metadata subsystem 140 in the centercolumn (e.g., using DD tracking and garbage collection logic 240 and anumber of data structures, e.g., 241, 242, 243), and operations at adata storage subsystem 150 in the right-hand column (e.g., using writelogic 252) adding to the global system deduplication virtual disk 1700,which is a replicated and partitioned virtual disk.

At block A, storage proxy 106 receives a data block (usrblock) targetinga user virtual disk 170 (e.g., usrvdisk_1). At block B, storage proxy106 calculates a hash value for the received data block. At block C,storage proxy 106 determines whether the calculated hash value is in theDDCache index 207. If yes, control passes to block L; otherwise, controlpasses to block D.

At block D, metadata subsystem 140 receives the hash value from storageproxy 106 and checks whether the hash value is in the DDInfo index 242.If yes, in the event that the metadata subsystem finds the hash valuereceived from the storage proxy in DDInfo 242, metadata subsystem 140responds to storage proxy 106 in the affirmative, providing theDDblockID and its associated expiry Ej according to DDInfo 242 and thencontrol passes to block J. In the event metadata subsystem 140 has norecord of the hash value in DDInfo 242, it responds in the negative tostorage proxy 106; in this case, storage proxy 106 causes the data blockto be added to the global system deduplication virtual disk 1700 atblock E.

At block E, data storage subsystem 150 writes the new data block(usrblock) to the global system deduplication virtual disk 1700 and anew and unique deduplication block identifier (DDblockID) is assigned tothe new data block at this point. Illustratively, 4 KB is the data blocksize stored at distributed data storage system 100 and hence eachDDblockID refers to a 4 KB data block. Henceforth, for simplicity, weshall refer to data blocks in distributed data storage system 100 usingthe term “DDblockID,” though it will be clear from context that the datablock is stored in its entirety in the global system deduplicationvirtual disk 1700 and is tracked or referenced elsewhere by itsDDblockID. Control then passes to blocks F and G. At block F, metadatasubsystem 140 updates the VdiskBlockInfo 241 column family and theDDTracker 243 column family to reflect the write request and theDDblockID being added to the distributed data storage system. At blockG, metadata subsystem 140 adds the new DDblockID to DDInfo 242 andassigns an expiry epoch (Ej) to DDblockID by adding 7 (as anillustrative example) to the value of the current epoch Enow. Theincrement of 7 is illustrative and stems from the fact that, becauseevery discard cycle increments the epoch by a total of 3, it isdesirable to give the data block at least two full discard cycles ofinitial lifetime (2×3=6). Since the timestamp epoch in DDTracker 243 isset to Enow+1, when 6 is added the increment becomes 7. Hence, at blockG, Ej=Enow+7. Metadata subsystem 140 transmits this information tostorage proxy 106. At block H, storage proxy 106 updates its DDCache 207by associating the hash value with the DDblockID and expiry epoch Ej. Atthis point, the I/O cycle for this data block is complete.

Block J is reached when storage proxy 106 receives a DDblockID andassociated expiry Ej from metadata subsystem 140 (from DDInfo 242) atblock D. At block J, storage proxy 106 determines whether the DDblockIDis expired according to the information received from metadata subsystem140, i.e., whether Ej is earlier than Enow. If the DDblockID is expired,storage proxy 106 treats the incoming data block as a new data block tobe added to the distributed data storage system and control passes toblock E, which is described in more detail above. On the other hand, ifat block J storage proxy 106 determines that the information receivedfrom DDInfo 242 indicates an unexpired DDblockID, the usrblock qualifiesfor deduplication and is not added to the global deduplication virtualdisk 1700. Instead, at block H, storage proxy 106 updates its DDCache207 by associating the hash value with the DDblockID and expiry epoch Ejreceived from DDInfo 242. Furthermore, at block F, metadata subsystem140 updates the VdiskBlockInfo 241 column family and the DDTracker 243column family to reflect the write request of the DDblockID. At thispoint, the I/O cycle for this data block is complete.

Block L is reached in the event that, at block C, storage proxy 106finds the calculated hash value of usrblock in its DDCache 207. InDDCache 207, the hash value is associated with a DDblockID having anexpiry epoch Ej and control passes to block L. At block L, storage proxy106 determines, based on DDCache 207, whether the DDblockID is expired,i.e., whether its expiry epoch Ej is earlier than Enow. If DDCache 207indicates that the DDblockID is expired, storage proxy 106 treats theincoming data block as if it weren't found in DDCache 207 and controlpasses to block D. Otherwise, if DDCache 207 indicates that theDDblockID is not expired, control passes to block M. Block M is the sameas block F, i.e., metadata subsystem 140 updates the VdiskBlockInfo 241column family and the DDTracker 243 column family to reflect the factthat a write request was received in the current epoch for thisDDblockID. At this point, the I/O cycle for usrblock ends with block M.

FIG. 4 depicts some salient operations of a method 400 according to anillustrative embodiment. Method 400 illustrates the first phase of adiscard cycle, the discard preparation (“GC prep”) stage. Method 400 isperformed by one or more components of the illustrative distributed datastorage system 100. The operations of metadata subsystem 140 in method400 are illustratively performed by DD tracking and garbage collectionlogic 240.

Block 401 depicts the scheme for setting the current epoch, Enow, to aidin the reader's understanding of the depicted method. In every discardcycle, Enow is incremented during the discard preparation stage (+2) andincremented again during every discard stage (+1). Accordingly, thenumerical values of Enow use the following pattern: Enow=0, 2(+2 GCPrep), 3(+1 GC), 5(+2 GC Prep), 6(+1 GC), 8, 9, etc. This pattern isshown at the top of FIG. 4 in block 401 and results from the operationat block 402. Illustratively, epoch calculator 230 tracks and calculatesEnow.

At block 402, at the beginning of a discard preparation stage, the podsubsystem (e.g., using epoch calculator 230) increments the currentepoch value by 2. The general formula is GC Prep Epoch=(3*N)−1, whereN≥1 and N is a discard cycle that includes the preparation stage and thediscard stage.

At block 404, a loop is initiated for each user (user-defined) virtualdisk configured on the distributed data storage system. The loopincludes blocks 406-410.

At block 406, on a first-time execution of the discard preparationstage, metadata subsystem 140 generates the data structure in FIG. 2F,e.g., column family 286, which comprises certain workload assignments.Accordingly, the workload of scanning DDTracker 243 column families (seeFIG. 2D) in each GC prep stage is assigned to a particular storageservice node 120/122. Thanks to consistent hashing, the primary metadatanode associated with each user virtual disk 170 is a deterministichostname that receives the workload assignment and will carry out thisworkload going forward. The present data structure 286 is persisted, sothat it can be used in the event that its host storage service nodefails.

At block 407, the metadata node assigns the scan workload to one or morestorage service nodes 120/122 according to data structure 286 in FIG.2F.

At block 408, the assigned storage service node(s) 120/122, using arespective metadata subsystem 140, scan the DDTracker 243 columnfamilies (generated in epoch Enow-2 or Enow-3, if any). Since a writerequest can come in at any time, such as during a preparation stage,this scheme ensures that all such write requests are scanned during thenext preparation stage. Thus, no write requests are left out ofDDTracker 243 and therefore no write requests are left unscanned. EachDDTracker 243 column family is associated with a particular user virtualdisk 170 (see FIG. 2D). The scan results are used to update DDRefCount244 column families. See also FIG. 2E. More details on block 408 aregiven in a subsequent figure.

At block 410, after the scanning task is completed, the DDTracker 243column families are discarded and control returns to block 404.

FIG. 5 depicts some salient operations of block 408 in method 400according to an illustrative embodiment. This block is performed by theprimary metadata node associated with each user virtual disk 170according to data structure 286 in FIG. 2F.

At block 502, a loop is initiated for each DDblockID in the DDTracker243 being scanned. The loop includes block 504.

At block 504, columns are added to DDRefCount 244 to: (i) populate theuser virtual disk 170 (uservdisk) column; (ii) from the timestamp entryin DDTracker 243, populate the epoch value associated with the timeframewhen the write request was received for the DDblockID, e.g., Ew; and(iii) assign an expiry epoch to the present reference based on the fullbackup frequency of the user virtual disk 170, e.g., Ew plus the fullbackup frequency of the user virtual disk 170. See also FIG. 2E. Controlpasses back to block 502.

At block 506, after DDRefCount 244 is fully updated from the DDTracker243 scans, DDRefCount 244 is persisted. DDRefCount 244 will be usedlater during the second phase of the discard cycle.

FIG. 6 depicts some salient operations of a method 600 according to anillustrative embodiment. Method 600 illustrates the second phase of thediscard cycle, the discard (“GC”) stage. Method 600 is performed by oneor more components of the illustrative distributed data storage system100 unless otherwise noted. The operations of metadata subsystem 140 inmethod 600 are illustratively performed by the DD tracking and garbagecollection logic 240.

Block 401 depicts the scheme for setting the current epoch, Enow, to aidin the reader's understanding of the depicted method. See also FIG. 4.

At block 602, at the beginning of a discard (GC) stage, pod subsystem130 (e.g., using epoch calculator 230) increments the current epoch Enowvalue by 1. The general formula is GC Epoch=(3*N), where N≥1 and N is adiscard cycle that includes the preparation stage and the discard stage.

At block 604, a loop is initiated for each storage container 260belonging to the global system deduplication virtual disk 1700. The loopincludes blocks 606-608, which are executed by metadata subsystem 140 inthe primary metadata node associated with the respective storagecontainer 260.

At block 606, metadata subsystem 140 scans DDRefCount 244 to determinewhether each DDblockID is both expired and has a zero reference count,in order to build the discard list 246 for the storage container 260.More details are given in a subsequent figure.

At block 608, the discard list 246 for the storage container 260 ispushed by metadata subsystem 140 to data storage subsystem 150 thathosts the storage container 260, possibly on another storage servicenode 120/122 distinct from the one hosting metadata subsystem 140. Atdata storage subsystem 150, a compaction process 250 will discard theDDblockIDs when it executes. In some embodiments, data storage subsystem150 pulls the container discard list(s) 246 from metadata subsystem 140asynchronously, when it has processing cycle time available. Thisalternative approach ensures that storage service nodes 120/122 do notget overwhelmed by the garbage cleanup task and instead can pull thediscard lists 246 whenever they deem fit. This approach also ensuresthat if a storage service node 120/122 is down when a discard list 246is pushed thereto, the storage service node can still obtain the discardlist 246 and purge the blocks later. See also FIG. 2B. Control passesback to block 604.

FIG. 7 depicts some salient operations of block 606 in method 600according to an illustrative embodiment. This block is performed by themetadata subsystem in the primary metadata node associated with therespective storage container.

At block 702, a loop is initiated for each DDblockID in the DDRefCount244 column family. The loop includes blocks 703-712.

At block 703, the reference count and/or entries in DDRefCount 244 aredecremented to account for the deletion of a user virtual disk 170 fromdistributed data storage system 100. Illustratively, each backup copyreceived by distributed data storage system 100 comprises one or morefiles, and the distributed data storage system addresses each file to afile-specific user virtual disk 170. When the backup system thatgenerated the backup copies prunes a stale backup copy from thedistributed data storage system, the result is that the distributed datastorage system deletes the various user virtual disks 170 associatedwith the stale backup copy. Thus, if user virtual disks 170 are deleted,they no longer reference certain DDblockIDs and this change is reflectedin DDRefCount 244 at this point. Therefore, it is possible that aDDblockID may reach a point when no user virtual disks 170 makereference to it anymore.

At block 704, metadata subsystem 140 determines whether there are anyreferences to the DDblockID in DDRefCount 244. As long there are, theDDblockID is not discarded.

At block 706, a maximum expiry epoch (e.g., Eu) is calculated for theDDblockID based on the reference expiry columns associated with the uservirtual disks 170 that reference the DDblockID. Alternatively, the Euvalue is extracted from DDRefCount 244, where Eu=Max(Ref Expirycolumns). See also FIG. 2E. At this point, metadata subsystem 140determines whether Eu is earlier than Enow-1. If so, the DDblockID ismarked expired.

At block 708, which is a decision point, if metadata subsystem 140determines that a given DDblockID is both expired (block 706) andcarries a zero reference count (block 704), control passes to block 710;otherwise control passes to block 709.

Block 709 is reached when a DDblockID does not meet the requirements forbeing discarded, i.e., it has a non-zero reference count and/or is notexpired. Here, the discard cycle considers whether an extension shouldbe added to the expiry of the present DDblockID. If a new write requestfor the present DDblockID came in after the preceding analysis cycle(i.e., after the last time this evaluation was made), the DDblockIDshould be extended, because it is still current and actively beingwritten by incoming backup copies. Accordingly, the Eu value fromDDRefCount 244 is now populated into the DDInfo index 242, replacing theexisting Ej expiry with the later Eu value (i.e., Eu>Ej). No replacementis needed or made if Ej≥Eu. Thus, here, a recently “written” data blockhas an opportunity for an extension to its expiry epoch long enough tospan the sparsest full backups of all the user virtual disks 170 stillreferencing the DDblockID. Control passes back to block 702.

Block 710 is reached when a DDblockID that is both expired (block 706)and has no valid reference counts from any user virtual disks 170 (block704). This DDblockID is now guaranteed to be suitable to discard fromdistributed data storage system 100.

At block 712, the DDblockID is added to the discard list 246 for thestorage container 260 being analyzed. See also FIG. 2G. Control passesback to block 702.

In regard to the figures described herein, other embodiments arepossible within the scope of the present invention, such that theabove-recited components, steps, blocks, operations, messages, requests,queries, and/or instructions are differently arranged, sequenced,sub-divided, organized, and/or combined. In some embodiments, adifferent component may initiate or execute a given operation.

Example Embodiments

Some example enumerated embodiments of the present invention are recitedin this section in the form of methods, systems, and non-transitorycomputer-readable media, without limitation.

According to an example embodiment, a distributed data storage systemcomprises: a storage proxy that executes on a first computing device; afirst storage service node that hosts a metadata subsystem; a secondstorage service node that stores a plurality of deduplicated datablocks, wherein a system-wide deduplication virtual disk comprises theplurality of deduplicated data blocks, and wherein the system-widededuplication virtual disk is distributed across a plurality of storageservice nodes of the distributed data storage system, including thesecond storage service node; wherein the storage proxy is configured to:intercept write requests addressed to one or more user virtual disksconfigured on the distributed data storage system, which are distinctfrom the system-wide deduplication virtual disk, wherein a first one ofthe write requests comprises a first data block addressed to a firstuser virtual disk, and cause the first data block to be stored in thesystem-wide deduplication virtual disk, at least at the second storageservice node; and wherein the metadata subsystem is configured to:assign an expiry timeframe to a first unique system-wide identifier (thefirst DDblockID) that is based on a hash value of and is associated withthe first data block, wherein the expiry timeframe is based at least inpart on an arrival timeframe of the first one of the write requests atthe storage proxy and is further based on a frequency of full backupoperations configured for the first user virtual disk, and cause thesecond storage service node to delete the first data block from thesystem-wide deduplication virtual disk, based on determining that (i) acurrent timeframe is later than the expiry timeframe of the firstDDblockID and (ii) no user virtual disk in the distributed data storagesystem makes reference to the first DDblockID.

The above-recited embodiment wherein the metadata subsystem is furtherconfigured to: cause the first data block to be deleted from thesystem-wide deduplication virtual disk, including from the secondstorage service node, even when second data blocks referenced by thefirst user virtual disk and associated with a second DDblockID, which isdistinct from the first DDblockID, are retained after the expirytimeframe of the first DDblockID. The above-recited embodimentconfigured with system-wide block-level deduplication and block-levelexpiry granularity. The above-recited embodiment wherein the expirytimeframe for the first DDblockID is further based on a maximum value of(i) the frequency of full backup operations configured for the firstuser virtual disk and (ii) one or more frequencies of full backupoperations corresponding to one or more other user virtual disksreferencing the first DDblockID. The above-recited embodiment whereinthe expiry timeframe is further based on a maximum value of (i) thefrequency of full backup operations configured for the first uservirtual disk and (ii) one or more frequencies of full backup operationscorresponding to one or more other user virtual disks referencing seconddata blocks having a same hash value as the first data block. Theabove-recited embodiment wherein the metadata subsystem is furtherconfigured to: update a data structure that tracks write requestsaddressed to the first user virtual disk, including the first one of thewrite requests that comprises the first data block. The above-recitedembodiment wherein the metadata subsystem is further configured to: if asecond data block received in a second one of the write requests isdetermined to have a same hash value as the first data block, isassociated with the first DDblockID, and arrived at the storage proxyafter a preceding discard cycle executed by the metadata subsystem,extend the expiry timeframe of the first DDblockID to span a sparsestfull backup frequency of all user virtual disks referencing the firstDDblockID. The above-recited embodiment wherein the metadata subsystemis further configured to: receive the hash value of the first data blockfrom the storage proxy, determine that the hash value is associated withthe first DDblockID, and update a data structure that tracks writerequests addressed to the first user virtual disk, including the firstone of the write requests that comprises the first data block. Theabove-recited embodiment wherein the first data block is associated withthe first DDblockID based on a hash value of the first data block;wherein the storage proxy is further configured to check whether theexpiry timeframe is less than a current timeframe; and wherein themetadata subsystem is further configured to: update a data structurethat tracks write requests addressed to the first user virtual disk,including the first one of the write requests that comprises the firstdata block. The above-recited embodiment wherein the first data block ispart of a backup copy, which is addressed to at least the first uservirtual disk. The above-recited embodiment wherein the first data blockis part of a backup copy addressed to the distributed data storagesystem; wherein when the backup copy is pruned, each distinct uservirtual disk configured for the backup copy is logically removed fromthe distributed data storage system, which causes references to datablocks of the backup copy to be removed from the distributed datastorage system; and wherein the metadata subsystem is further configuredto: retain a second data block supplied by the backup copy if at leastone other user virtual disk, not associated with the backup copy,references a DDblockID associated with the second data block.

According to another example embodiment, a distributed data storagesystem for storing backup copies with deduplication comprises: a firststorage service node that hosts a metadata subsystem; a second storageservice node that stores deduplicated data blocks, wherein a system-widededuplication virtual disk is distributed across a plurality of storageservice nodes of the distributed data storage system, including thesecond storage service node, wherein the system-wide deduplicationvirtual disk comprises the deduplicated data blocks, and wherein eachdeduplicated data block in the system-wide deduplication virtual disk isassociated with a corresponding unique system-wide identifier; whereinthe metadata subsystem is configured to: track data blocks addressed touser virtual disks configured in the distributed data storage system,including a first data block in a first write request addressed to afirst user virtual disk, which is distinct from the system-widededuplication virtual disk; associate the first data block with a firstunique system-wide identifier (the first DDblockID) based on a hashvalue of the first data block, assign an expiry timeframe to the firstDDblockID, wherein the expiry timeframe is based at least in part on anarrival timeframe of the first write request at a storage proxy and isfurther based on a frequency of full backup operations configured forthe first user virtual disk, for a second data block, which arrived in asecond write request after a preceding discard cycle executed by themetadata subsystem, wherein the second data block has a same hash valueas the first data block which is associated with the first DDblockID,extend the expiry timeframe of the first DDblockID to span a sparsestfull backup frequency of all user virtual disks referencing the firstDDblockID.

The above-recited embodiment wherein the metadata subsystem is furtherconfigured to: cause a data block having the hash value of the firstdata block and associated with the first DDblockID to be retained in thesystem-wide deduplication virtual disk, based on determining that atleast one of: (i) a current timeframe is earlier than the expirytimeframe of the first DDblockID and (ii) at least one user virtual diskin the distributed data storage system makes reference to the firstDDblockID. The above-recited embodiment wherein the metadata subsystemis further configured to: cause a data block having the hash value ofthe first data block and associated with the first DDblockID to bedeleted from the system-wide deduplication virtual disk, including fromthe second storage service node, based on determining that: (a) acurrent timeframe is later than the expiry timeframe of the firstDDblockID and (b) no user virtual disk on the distributed data storagesystem makes reference to the first DDblockID. The above-recitedembodiment wherein the first user virtual disk is configured fordeduplication using block-level expiry granularity. The above-recitedembodiment wherein the expiry timeframe is further based on a maximumvalue of (i) the frequency of full backup operations configured for thefirst user virtual disk and (ii) one or more frequencies of full backupoperations corresponding to one or more other user virtual disksreferencing the first DDblockID. The above-recited embodiment whereinthe expiry timeframe is further based on a maximum value of (i) thefrequency of full backup operations configured for the first uservirtual disk and (ii) one or more frequencies of full backup operationscorresponding to one or more other user virtual disks referencing seconddata blocks having a same hash value as the first data block. Theabove-recited embodiment wherein the storage proxy is further configuredto check whether the expiry timeframe is less than the currenttimeframe. The above-recited embodiment wherein the first data block ispart of a backup copy, which is addressed to at least the first uservirtual disk. The above-recited embodiment wherein the first data blockis part of a backup copy addressed to the distributed data storagesystem, wherein pruning the backup copy causes references to data blocksof the backup copy to be removed from the distributed data storagesystem; and wherein the metadata subsystem is further configured to:retain a second data block supplied by the backup copy if at least oneother user virtual disk, not associated with the backup copy, referencesa DDblockID associated with the second data block.

According to an illustrative embodiment, a method for providingdeduplication with block-level expiry granularity in a cloud-baseddistributed data storage system comprises: in a first cloud computingenvironment, configuring a first storage service node that hosts ametadata subsystem of the distributed data storage system, and furtherconfiguring a second storage service node that hosts a data storagesubsystem of the distributed data storage subsystem, wherein thedistributed data storage subsystem comprises a system-wide deduplicationvirtual disk that is distributed across a plurality of storage servicenodes of the distributed data storage system, including the secondstorage service node, and wherein the system-wide deduplication virtualdisk stores deduplicated data blocks; by the first storage service nodehosting the metadata subsystem, assigning an expiry timeframe to a firstunique system-wide identifier (the first DDblockID) that is based on ahash value of, and is associated with, a first data block stored in thesystem-wide deduplication virtual disk, wherein the expiry timeframe isbased at least in part on: an arrival timeframe of the first data blockat the distributed data storage system, and is further based on a fullbackup frequency configured for a first user virtual disk addressed by awrite request comprising the first data block; and by the first storageservice node hosting the metadata subsystem, if a second data block,which is received in a second write request addressing a second uservirtual disk after a preceding discard cycle executed by the metadatasubsystem, is determined to have a same hash value as the first datablock and is associated with the first DDblockID, extending the expirytimeframe of the first DDblockID to span a sparsest full backupfrequency of all user virtual disks referencing the first DDblockID,including the first user virtual disk and the second user virtual disk.

The above-recited embodiment further comprising: by the first storageservice node hosting the metadata subsystem, causing the second storageservice node to delete the first data block from the system-widededuplication virtual disk, based on determining that (i) a currenttimeframe is later than the expiry timeframe of the first DDblockID and(ii) no user virtual disk in the distributed data storage system makesreference to the first DDblockID. The above-recited embodiment furthercomprising: by the second storage service node hosting the data storagesubsystem, deleting the first data block from the system-widededuplication virtual disk, based on a determination by the metadatasubsystem that (i) a current timeframe is later than the expirytimeframe of the first DDblockID and (ii) no user virtual disk in thedistributed data storage system makes reference to the first DDblockID.The above-recited embodiment further comprising: by a compaction logicof the data storage subsystem, deleting the first data block from thesystem-wide deduplication virtual disk, based on a determination by themetadata subsystem that (i) a current timeframe is later than the expirytimeframe of the first DDblockID and (ii) no user virtual disk in thedistributed data storage system makes reference to the first DDblockID.The above-recited embodiment wherein the first data block is part of abackup copy addressed to the distributed data storage system, whereinpruning the backup copy causes references to data blocks of the backupcopy to be removed from the distributed data storage system; andretaining a third data block supplied by the backup copy if at least oneother user virtual disk, not associated with the backup copy, referencesa DDblockID associated with the third data block. The above-recitedembodiment wherein the first data block is part of a backup copyaddressed to the distributed data storage system; wherein when thebackup copy is pruned, each distinct user virtual disk configured forthe backup copy is logically removed from the distributed data storagesystem, which causes references to data blocks of the backup copy to beremoved from the distributed data storage system; and by the metadatasubsystem causing a third data block supplied by the backup copy to beretained in the system-wide deduplication virtual disk if at least oneother user virtual disk, not associated with the backup copy, referencesa DDblockID associated with the third data block. The above-recitedembodiment wherein the first data block is part of a backup copyaddressed to the distributed data storage system, and wherein the backupcopy is generated in one of: within the first cloud computingenvironment, and outside the first cloud computing environment. Theabove-recited embodiment wherein the expiry timeframe for the firstDDblockID is further based on a maximum value of (i) the full backupfrequency configured for the first user virtual disk and (ii) one ormore full backup frequencies corresponding to one or more other uservirtual disks referencing the first DDblockID. The above-recitedembodiment wherein the expiry timeframe is further based on a maximumvalue of (i) the full backup frequency configured for the first uservirtual disk and (ii) one or more full backup frequencies correspondingto one or more other user virtual disks referencing second data blockshaving a same hash value as the first data block. The above-recitedembodiment further comprising: by a storage proxy that executes on afirst computing device that is operational outside the first cloudcomputing environment: intercepting write requests addressed to one ormore user virtual disks configured on the distributed data storagesystem, which are distinct from the system-wide deduplication virtualdisk, including intercepting the write request comprising the first datablock, and causing the first data block to be stored in the system-widededuplication virtual disk, at least at the second storage service node.The above-recited embodiment further comprising: by a storage proxy thatexecutes on a first computing device that is operational within thefirst cloud computing environment: intercepting write requests addressedto one or more user virtual disks configured on the distributed datastorage system, which are distinct from the system-wide deduplicationvirtual disk, including intercepting the write request comprising thefirst data block, and causing the first data block to be stored in thesystem-wide deduplication virtual disk, at least at the second storageservice node. The above-recited embodiment further comprising: by astorage proxy that executes on a first computing device, interceptingwrite requests addressed to one or more user virtual disks configured onthe distributed data storage system, which are distinct from thesystem-wide deduplication virtual disk, including intercepting the writerequest comprising the first data block; and by the metadata subsystem:receiving the hash value of the first data block from the storage proxy,determining that the hash value is associated with the first DDblockID,and updating a data structure that tracks write requests addressed tothe first user virtual disk, including the write request that comprisesthe first data block. The above-recited embodiment further comprising:by a storage proxy that executes on a first computing device,intercepting write requests addressed to one or more user virtual disksconfigured on the distributed data storage system, which are distinctfrom the system-wide deduplication virtual disk, including interceptingthe write request comprising the first data block; and by the metadatasubsystem: receiving the hash value of the first data block from thestorage proxy, determining that the hash value is associated with thefirst DDblockID, and updating a data structure that tracks writerequests addressed to the first user virtual disk, including the writerequest that comprises the first data block.

According to another illustrative embodiment, a method for providingdeduplication with block-level expiry granularity in a distributed datastorage system comprises: in a first cloud computing environmentcomprising a first storage service node that hosts a metadata subsystemof the distributed data storage system, and further comprising a secondstorage service node that hosts a data storage subsystem of thedistributed data storage subsystem, wherein the distributed data storagesubsystem is configured with a system-wide deduplication virtual diskthat is distributed across a plurality of storage service nodes of thedistributed data storage system, including the second storage servicenode, and wherein the system-wide deduplication virtual disk storesdeduplicated data blocks: by the metadata subsystem at first storageservice node, assigning an expiry timeframe to a first uniquesystem-wide identifier (the first DDblockID), wherein the firstDDblockID is based on a hash value of, and is associated with, a firstdata block stored in the system-wide deduplication virtual disk, whereinthe expiry timeframe is based at least in part on: an arrival timeframeof the first data block at the distributed data storage system, andspans a full backup frequency configured for a first user virtual diskaddressed by a write request comprising the first data block, whereinthe first user virtual disk is distinct from the system-widededuplication virtual disk; and by the metadata subsystem executing adiscard cycle: determining that a second data block, which was receivedin a second write request addressing a second user virtual disk andarrived at the distributed data storage system after a preceding discardcycle has a same hash value as the first data block, associating thesecond data block with the first DDblockID based on the same hash value,and extending the expiry timeframe of the first DDblockID to span asparsest full backup frequency of all user virtual disks referencing thefirst DDblockID, including the first user virtual disk and the seconduser virtual disk.

The above-recited embodiment wherein the assigning of the expirytimeframe occurs during an input-output cycle of the first data block,which is distinct from the discard cycle. The above-recited embodimentfurther comprising: by the metadata subsystem executing a discard cycle,causing the second storage service node to delete the first data blockfrom the system-wide deduplication virtual disk, based on the metadatasubsystem determining that (i) a current timeframe is later than theexpiry timeframe of the first DDblockID and (ii) no user virtual disk inthe distributed data storage system makes reference to the firstDDblockID. The above-recited embodiment further comprising: by thesecond storage service node hosting the data storage subsystem, deletingthe first data block from the system-wide deduplication virtual disk,based on a determination by the metadata subsystem that (i) a currenttimeframe is later than the expiry timeframe of the first DDblockID and(ii) no user virtual disk in the distributed data storage system makesreference to the first DDblockID. The above-recited embodiment whereinthe first data block is part of a backup copy addressed to thedistributed data storage system, wherein pruning the backup copy causesreferences to data blocks of the backup copy to be removed from thedistributed data storage system; and retaining a third data blocksupplied by the backup copy if at least one other user virtual disk, notassociated with the backup copy, references a DDblockID associated withthe third data block. The above-recited embodiment wherein the firstdata block is part of a backup copy addressed to the distributed datastorage system, and wherein the backup copy is generated in one of:within the first cloud computing environment, and outside the firstcloud computing environment. The above-recited embodiment furthercomprising: by a storage proxy that executes on a first computing devicethat is one of: operational outside the first cloud computingenvironment, and operational within the first cloud computingenvironment: intercepting write requests addressed to one or more uservirtual disks configured on the distributed data storage system, whichare distinct from the system-wide deduplication virtual disk, includingintercepting the write request comprising the first data block, andcausing the first data block to be stored in the system-widededuplication virtual disk, at least at the second storage service node.

According to another illustrative embodiment, a system comprises adistributed data storage platform having system-wide deduplication withblock-level expiry granularity. The above-recited embodiment wherein theuseful life of each deduplicated data block is based on expiryparameters that relate to backup frequencies of the virtual disksreferencing the data block, thus guaranteeing that data blocks are keptaround between full backup cycles and are extended if still current. Theabove-recited embodiment wherein data blocks are retained as long asneeded to bridge the gap between sparser backup operations. Theabove-recited embodiment wherein tracking data structures are updatedonly as needed, thus saving processing cycles and network bandwidth. Theabove-recited embodiment wherein the distributed data storage platformguarantees that stale references to DDblockIDs lingering innon-functional components cannot dictate whether a particular DDblockIDis discarded.

In other embodiments according to the present invention, a system orsystems operates according to one or more of the methods and/orcomputer-readable media recited in the preceding paragraphs. In yetother embodiments, a method or methods operates according to one or moreof the systems and/or computer-readable media recited in the precedingparagraphs. In yet more embodiments, a non-transitory computer-readablemedium or media causes one or more computing devices having one or moreprocessors and computer-readable memory to operate according to one ormore of the systems and/or methods recited in the preceding paragraphs.

Terminology

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense, i.e., in the sense of “including, but notlimited to.” As used herein, the terms “connected,” “coupled,” or anyvariant thereof means any connection or coupling, either direct orindirect, between two or more elements; the coupling or connectionbetween the elements can be physical, logical, or a combination thereof.Additionally, the words “herein,” “above,” “below,” and words of similarimport, when used in this application, refer to this application as awhole and not to any particular portions of this application. Where thecontext permits, words using the singular or plural number may alsoinclude the plural or singular number respectively. The word “or” inreference to a list of two or more items, covers all of the followinginterpretations of the word: any one of the items in the list, all ofthe items in the list, and any combination of the items in the list.Likewise the term “and/or” in reference to a list of two or more items,covers all of the following interpretations of the word: any one of theitems in the list, all of the items in the list, and any combination ofthe items in the list.

In some embodiments, certain operations, acts, events, or functions ofany of the algorithms described herein can be performed in a differentsequence, can be added, merged, or left out altogether (e.g., not allare necessary for the practice of the algorithms). In certainembodiments, operations, acts, functions, or events can be performedconcurrently, e.g., through multi-threaded processing, interruptprocessing, or multiple processors or processor cores or on otherparallel architectures, rather than sequentially.

Systems and modules described herein may comprise software, firmware,hardware, or any combination(s) of software, firmware, or hardwaresuitable for the purposes described. Software and other modules mayreside and execute on servers, workstations, personal computers,computerized tablets, PDAs, and other computing devices suitable for thepurposes described herein. Software and other modules may be accessiblevia local computer memory, via a network, via a browser, or via othermeans suitable for the purposes described herein. Data structuresdescribed herein may comprise computer files, variables, programmingarrays, programming structures, or any electronic information storageschemes or methods, or any combinations thereof, suitable for thepurposes described herein. User interface elements described herein maycomprise elements from graphical user interfaces, interactive voiceresponse, command line interfaces, and other suitable interfaces.

Further, processing of the various components of the illustrated systemscan be distributed across multiple machines, networks, and othercomputing resources. Two or more components of a system can be combinedinto fewer components. Various components of the illustrated systems canbe implemented in one or more virtual machines, rather than in dedicatedcomputer hardware systems and/or computing devices. Likewise, the datarepositories shown can represent physical and/or logical data storage,including, e.g., storage area networks or other distributed storagesystems. Moreover, in some embodiments the connections between thecomponents shown represent possible paths of data flow, rather thanactual connections between hardware. While some examples of possibleconnections are shown, any of the subset of the components shown cancommunicate with any other subset of components in variousimplementations.

Embodiments are also described above with reference to flow chartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products. Each block of the flow chart illustrationsand/or block diagrams, and combinations of blocks in the flow chartillustrations and/or block diagrams, may be implemented by computerprogram instructions. Such instructions may be provided to a processorof a general purpose computer, special purpose computer,specially-equipped computer (e.g., comprising a high-performancedatabase server, a graphics subsystem, etc.) or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor(s) of the computer or other programmabledata processing apparatus, create means for implementing the actsspecified in the flow chart and/or block diagram block or blocks. Thesecomputer program instructions may also be stored in a non-transitorycomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to operate in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meanswhich implement the acts specified in the flow chart and/or blockdiagram block or blocks. The computer program instructions may also beloaded to a computing device or other programmable data processingapparatus to cause operations to be performed on the computing device orother programmable apparatus to produce a computer implemented processsuch that the instructions which execute on the computing device orother programmable apparatus provide steps for implementing the actsspecified in the flow chart and/or block diagram block or blocks.

Any patents and applications and other references noted above, includingany that may be listed in accompanying filing papers, are incorporatedherein by reference. Aspects of the invention can be modified, ifnecessary, to employ the systems, functions, and concepts of the variousreferences described above to provide yet further implementations of theinvention. These and other changes can be made to the invention in lightof the above Detailed Description. While the above description describescertain examples of the invention, and describes the best modecontemplated, no matter how detailed the above appears in text, theinvention can be practiced in many ways. Details of the system may varyconsiderably in its specific implementation, while still beingencompassed by the invention disclosed herein. As noted above,particular terminology used when describing certain features or aspectsof the invention should not be taken to imply that the terminology isbeing redefined herein to be restricted to any specific characteristics,features, or aspects of the invention with which that terminology isassociated. In general, the terms used in the following claims shouldnot be construed to limit the invention to the specific examplesdisclosed in the specification, unless the above Detailed Descriptionsection explicitly defines such terms. Accordingly, the actual scope ofthe invention encompasses not only the disclosed examples, but also allequivalent ways of practicing or implementing the invention under theclaims.

To reduce the number of claims, certain aspects of the invention arepresented below in certain claim forms, but the applicant contemplatesother aspects of the invention in any number of claim forms. Forexample, while only one aspect of the invention is recited as ameans-plus-function claim under 35 U.S.C sec. 112(f) (AIA), otheraspects may likewise be embodied as a means-plus-function claim, or inother forms, such as being embodied in a computer-readable medium. Anyclaims intended to be treated under 35 U.S.C. § 112(f) will begin withthe words “means for,” but use of the term “for” in any other context isnot intended to invoke treatment under 35 U.S.C. § 112(f). Accordingly,the applicant reserves the right to pursue additional claims afterfiling this application, in either this application or in a continuingapplication.

What is claimed is:
 1. A computer-implemented method comprising: by afirst computing device, which comprises one or more hardware processorsand data storage resources, wherein the first computing device isconfigured to execute a storage proxy: intercepting a first writerequest addressed to a first user virtual disk configured on adistributed data storage system, wherein the first user virtual disk isdistinct from a deduplication virtual disk, wherein the first writerequest comprises a first data block addressed to the first user virtualdisk, and causing the first data block to be stored in the deduplicationvirtual disk, at data storage resources of least a second storageservice node, wherein the second storage service node comprises one ormore hardware processors and data storage resources, wherein the secondstorage service node is configured to execute a data storage subsystem;and by a first storage service node, which comprises one or morehardware processors and data storage resources, wherein the firststorage service node is configured to execute a metadata subsystem:assigning an expiry timeframe to a first unique system-wide identifierthat is based on a hash value of, and is associated with, the first datablock, wherein the expiry timeframe is based at least in part on anarrival timeframe of the first write request at the storage proxy and isfurther based on a frequency of full-backup operations configured forthe first user virtual disk, and causing the second storage service nodeto delete the first data block from the deduplication virtual disk,based on determining that (i) a current timeframe is later than theexpiry timeframe of the first unique system-wide identifier and (ii) nouser virtual disk in the distributed data storage system makes referenceto the first unique system-wide identifier.
 2. The computer-implementedmethod of claim 1, further comprising: causing, by the first storageservice node, the first data block to be deleted from the deduplicationvirtual disk, including from the second storage service node, even whensecond data blocks referenced by the first user virtual disk andassociated with a second unique system-wide identifier, which isdistinct from the first unique system-wide identifier, are retainedafter the expiry timeframe of the first unique system-wide identifier.3. The computer-implemented method of claim 1, wherein a distributeddata storage system, which comprises the first storage service node andthe second storage service node is configured with block-leveldeduplication and block-level expiry granularity, wherein theblock-level expiry granularity enables the distributed data storagesystem to delete expired data blocks from the deduplication virtual diskregardless of whether other data blocks in the deduplication virtualdisk, distinct from the expired data blocks, are referenced by a uservirtual disk that also referenced the expired data blocks.
 4. Thecomputer-implemented method of claim 1, wherein the expiry timeframe forthe first unique system-wide identifier is further based on a maximumvalue that spans a sparsest frequency among one or more frequencies offull-backup operations configured for all user virtual disks thatreference the first unique system-wide identifier, including the firstuser virtual disk.
 5. The computer-implemented method of claim 1 furthercomprising: based on determining, by the first storage service node,that a second data block received in a second write request has a samehash value as the first data block, associating the second data blockwith the first unique system-wide identifier; based on determining, bythe first storage service node, that the second data block arrived atthe storage proxy after a preceding discard cycle was executed by themetadata subsystem, extending the expiry timeframe of the first uniquesystem-wide identifier to span a sparsest frequency, among one or morefrequencies of full-backup operations respectively configured for one ormore user virtual disks, including for the first user virtual disk, thatreference the first unique system-wide identifier.
 6. Thecomputer-implemented method of claim 1, further comprising: receiving,by the first storage service node, the hash value of the first datablock from the storage proxy, determining, by the first storage servicenode, that the hash value of the first data block is associated with thefirst unique system-wide identifier, and updating, by the first storageservice node, a data structure that tracks write requests addressed tothe first user virtual disk, including the first write request thatcomprises the first data block.
 7. The computer-implemented method ofclaim 1, wherein the first data block is part of a backup copy stored inthe distributed data storage system; and further comprising: when thebackup copy is pruned from the distributed data storage system,logically removing each user virtual disk configured for the backup copyfrom the distributed data storage system, which causes references todata blocks of the backup copy to be removed from the distributed datastorage system; and based on determining, by the first storage servicenode, that at least one user virtual disk, within the distributed datastorage system and not associated with the backup copy, references thefirst unique system-wide identifier associated with the first datablock, retaining a reference, by the at least one user virtual disk notassociated with the backup copy, to the first unique system-wideidentifier associated with the first data block even though the firstdata block is part of the backup copy being pruned.
 8. A systemcomprising: a first storage service node, which comprises one or morehardware processors and data storage resources, wherein the firststorage service node is configured to execute a metadata subsystem; anda second storage service node, which comprises one or more hardwareprocessors and data storage resources, wherein the second storageservice node is configured to execute a data storage subsystem; whereinthe system is configured with a deduplication virtual disk, which storesdata blocks, in deduplicated form, in data storage resources among aplurality of storage service nodes of the system, including at leastsome of the data storage resources of the second storage service node,and wherein the system is further configured with at least a first uservirtual disk to which some write requests are addressed comprising datablocks that are to be stored in the system; and wherein the firststorage service node hosting the metadata subsystem is configured to:assign a unique system-wide identifier to each data block that is storedwith deduplication in the deduplication virtual disk, based on a hashvalue respectively corresponding to each data block, adjust a givenexpiry timeframe corresponding to a given unique system-wide identifierof a given data block stored in the deduplication virtual disk, whereinthe given expiry timeframe is adjusted to span a sparsest frequency,selected from among one or more frequencies of full-backup operationsrespectively corresponding to one or more user virtual disks configuredin the system, that reference the given unique system-wide identifier,and cause the system to delete from the deduplication virtual disk afirst data block, which was assigned a first unique system-wideidentifier, based on determining that (i) a current timeframe is laterthan a first expiry timeframe of the first unique system-wideidentifier, and (ii) no user virtual disk configured in the system makesreference to the first unique system-wide identifier.
 9. The system ofclaim 8, wherein the first storage service node hosting the metadatasubsystem is further configured to: (a) assign the first expirytimeframe to the first unique system-wide identifier that is associatedwith the first data block, wherein the first expiry timeframe is basedon an arrival timeframe of the first data block at the system, wherein afirst write request comprising the first data block is addressed to thefirst user virtual disk, and wherein the first expiry timeframe isfurther based on a first frequency of full-backup operationscorresponding to the first user virtual disk, and (b) based on a secondwrite request that comprises a second data block with a same hash valueas the first data block, wherein the second write request is addressedto a second user virtual disk configured in the system, and wherein thesecond write request is received after a preceding discard cycle wasexecuted by the metadata subsystem: associate the second data block withthe first unique system-wide identifier, based on having the same hashvalue as the first data block, and adjust the first expiry timeframe ofthe first unique system-wide identifier to span a sparsest frequency,selected from one or more frequencies of full-backup operationsrespectively corresponding to one or more user virtual disks configuredin the system that reference the first unique system-wide identifier,including the first frequency corresponding to the first user virtualdisk and a second frequency corresponding to the second user virtualdisk.
 10. The system of claim 8, wherein the first data block in thededuplication virtual disk is part of a backup copy stored in thesystem; and wherein the system is further configured to: when the backupcopy is pruned from the system, logically remove from the system eachdistinct user virtual disk configured for the backup copy, which causesreferences to data blocks of the backup copy to be removed from thesystem; and based on determining, by the first storage service nodehosting the metadata subsystem, that at least one user virtual disk notassociated with the backup copy references the first unique system-wideidentifier associated with the first data block, retaining a reference,by the at least one user virtual disk not associated with the backupcopy, to the first unique system-wide identifier associated with thefirst data block even though the first data block is part of the backupcopy being pruned.
 11. A system comprising: a first storage servicenode, which comprises one or more hardware processors and data storageresources, wherein the first storage service node is configured toexecute a metadata subsystem; and a second storage service node, whichcomprises one or more hardware processors and data storage resources,wherein the second storage service node is configured to execute a datastorage subsystem; wherein the system is configured with a deduplicationvirtual disk that stores data blocks, in deduplicated form, in datastorage resources among a plurality of storage service nodes of thesystem, including at least some of the data storage resources of thesecond storage service node, wherein the system does not expose thededuplication virtual disk as a storage target for addressing writerequests, and wherein the system is further configured with at least afirst user virtual disk to which write requests are addressed for datablocks to be stored in the system; wherein the first storage servicenode hosting the metadata subsystem is configured to: (a) assign anexpiry timeframe to a first unique system-wide identifier that isassociated with a first data block that is stored in the deduplicationvirtual disk, wherein the expiry timeframe is based on an arrivaltimeframe of the first data block at the system, and is further based ona frequency of full-backup operations corresponding to the first uservirtual disk, wherein a first write request comprising the first datablock is addressed to the first user virtual disk, and (b) based on asecond write request that comprises a second data block with a same hashvalue as the first data block, and wherein the second write request isaddressed to a second user virtual disk configured in the system, andwherein the second write request is received after a preceding discardcycle was executed by the metadata subsystem: associate the second datablock with the first unique system-wide identifier, based on the samehash value, and set the expiry timeframe of the first unique system-wideidentifier to a maximum value of (i) the frequency of full-backupoperations corresponding to the first user virtual disk, (ii) afrequency of full-backup operations corresponding to the second uservirtual disk, and (iii) one or more frequencies of full-backupoperations respectively corresponding to one or more other user virtualdisks that reference the first unique system-wide identifier.
 12. Thesystem of claim 11, wherein the first storage service node hosting themetadata subsystem is further configured to: cause the second storageservice node to delete the first data block from the deduplicationvirtual disk, based on determining that (i) a current timeframe is laterthan the expiry timeframe of the first unique system-wide identifier,and (ii) no user virtual disk configured in the system makes referenceto the first unique system-wide identifier.
 13. The system of claim 11,wherein the second storage service node hosting the data storagesubsystem is configured to: delete the first data block from thededuplication virtual disk, based on a determination by the metadatasubsystem that (i) a current timeframe is later than the expirytimeframe of the first unique system-wide identifier, and (ii) no uservirtual disk configured in the system makes reference to the firstunique system-wide identifier.
 14. The system of claim 11, wherein thesystem is configured to delete references to data blocks that are partof any backup copy being pruned from the system, and wherein the systemis further configured to: determine that the first data block is part ofa first backup copy stored in the system; and retain the first datablock in the deduplication virtual disk as part of pruning the firstbackup copy from the system, based on determining that at least one uservirtual disk, which is not associated with the first backup copy,references the first unique system-wide identifier associated with thefirst data block.
 15. The system of claim 14, wherein the first storageservice node hosting the metadata subsystem is further configured to:determine that at least one user virtual disk, which is not associatedwith the first backup copy, references the first unique system-wideidentifier, which is associated with the first data block; and cause thesystem to retain the first data block in the deduplication virtual diskas part of pruning the first backup copy from the system.
 16. The systemof claim 11, wherein the first storage service node and the secondstorage service node operate in a first cloud computing environment, andfurther wherein the first data block is part of a backup copy stored inthe system, and further wherein the backup copy is generated in one of:within the first cloud computing environment, and outside the firstcloud computing environment.
 17. The system of claim 11, wherein thefirst storage service node hosting the metadata subsystem is furtherconfigured to: for a given discard cycle, maintain a first datastructure that tracks, for every user virtual disk in the system, everynew write request received by the system in the given discard cycle,wherein, for a given user virtual disk, the first data structureidentifies one or more of: a unique system-wide identifier associatedwith a given data block stored in the system, a hash value for the givendata block, and a timestamp based on an arrival timeframe at the system,within the given discard cycle, of a data block having the hash value.18. The system of claim 17, wherein the first storage service nodehosting the metadata subsystem is further configured to, after apreceding discard cycle was executed by the metadata subsystem: scan thefirst data structure; for a given unique system-wide identifier, updatea second data structure, based on information in the first datastructure, wherein the second data structure indicates which uservirtual disk in the system references the given unique system-wideidentifier, and further indicates an expiry timeframe of the givenunique system-wide identifier that is a maximum value of one or morefrequencies of full-backup operations corresponding to any user virtualdisks in the system that reference the given unique system-wideidentifier; and discard the first data structure.
 19. The system ofclaim 11, wherein the expiry timeframe of the first unique system-wideidentifier, which is associated with the first data block that is storedin the deduplication virtual disk, provides the system with block-levelexpiry granularity for deleting expired data blocks from thededuplication virtual disk, regardless of whether other data blocks inthe deduplication virtual disk, distinct from the expired data blocks,are referenced by a user virtual disk that referenced the expired datablocks.
 20. The system of claim 11, wherein the first storage servicenode hosting the metadata subsystem is configured to: associate a givenunique data block, which is stored with deduplication in thededuplication virtual disk, with a corresponding expiry timeframe thatis based on (A) an arrival time at the system of the given unique datablock and (B) respective arrival times of any other data blocks with asame hash value as the given unique data block, and wherein thecorresponding expiry timeframe is further based on frequencies offull-backup operations respectively corresponding to one or more uservirtual disks that reference the given unique data block.